diff --git a/README.md b/README.md index b9768bb..8681d03 100644 --- a/README.md +++ b/README.md @@ -1,12 +1,48 @@ # Benchmarks -Community benchmarks for agentgateway. +Community benchmarks for [agentgateway](https://github.com/agentgateway/agentgateway). -- [`inference/`](inference/README.md) — inference benchmark execution, results, and publishable reports (plain Kubernetes Service vs. agentgateway standalone vs. agentgateway on Kubernetes, run across GKE campaigns) +This repo holds the tooling and published results for measuring how agentgateway +performs as an inference gateway - both as EPP's standalone sidecar and as a +Gateway API data plane - compared to routing traffic through a plain Kubernetes +Service with no gateway at all. -PR #1 intentionally migrates the inference benchmark tooling only. Published -benchmark reports and their supporting evidence will follow separately after -the tooling migration is complete. +## Layout -Automated repository checks cover syntax and unit tests only; they do not -provision infrastructure or execute benchmark campaigns. +- [`inference/`](inference/README.md) - the benchmark runner itself: campaign-based + execution comparing `service`, `agentgateway-standalone`, and + `agentgateway-gateway` treatments, with automated GKE provisioning/teardown and + Markdown/PNG/CSV report generation. +- [`inference/reports/`](inference/reports/README.md) - curated, published benchmark + results with their campaign manifests and provenance, safe to link to directly. + +## Quick start + +Run the `service` treatment locally on Kind: + +```bash +make kind-create +BENCHMARK_TREATMENT=service BENCHMARK_CAMPAIGN_ID=local-sim make benchmark +``` + +The `service` treatment runs fine on a laptop-sized Kind cluster. +`agentgateway-standalone` and `agentgateway-gateway` currently need more CPU/memory +than most laptops provide for the router pod - see +[#2](https://github.com/agentgateway/benchmarks/issues/2). + +See [`inference/README.md`](inference/README.md) for the full set of treatments, +GKE campaign instructions, and configuration reference. + +## Current results + +The most recent published campaign +([`optimized-baseline-v0230-gateway-refresh-20260817`](inference/reports/llm-d-benchmark/optimized-baseline-qwen3-32b-h100/optimized-baseline-v0230-gateway-refresh-20260817/README.md)) +ran Qwen/Qwen3-32B on 16x H100 GPUs across 8 vLLM model servers. At the top of the +request-rate ladder, agentgateway more than doubles peak throughput and cuts TTFT +p90 by over 99% versus a plain Kubernetes Service doing round-robin, because +round-robin has no way to know which backend pod is already overloaded and +agentgateway's EPP-based routing does. + +Automated repository checks (`inference/suites/*/tests`, `inference/reporting/tests`) +cover syntax and unit tests only - they don't provision infrastructure or execute +benchmark campaigns. diff --git a/inference/README.md b/inference/README.md index f3f58c2..8724691 100644 --- a/inference/README.md +++ b/inference/README.md @@ -5,10 +5,6 @@ retains their native evidence in a shared campaign, and generates publishable Markdown, PNG, and CSV comparisons. Prism-compatible Benchmark Report v0.2 documents remain available in each treatment's evidence bundle. -PR #1 intentionally migrates tooling only. Curated reports from the original -agentgateway repository change are not part of this migration and will be -published separately with their supporting evidence. - ## Layout ```text diff --git a/inference/reports/README.md b/inference/reports/README.md index c716049..32dfefc 100644 --- a/inference/reports/README.md +++ b/inference/reports/README.md @@ -1,14 +1,12 @@ -# Published inference reports +# Published inference benchmark reports -PR #1 intentionally migrates the inference benchmark tooling only. Published -reports and their supporting evidence will be added separately after the -tooling migration is complete. - -Curated reports belong under a suite and profile-specific path, for example: +Curated benchmark reports may be committed below this directory using: ```text -inference/reports/llm-d-benchmark/optimized-baseline-qwen3-32b-h100/ +/// ``` -Generated local reports remain under the ignored campaign directory in -`inference/results/` until they are reviewed and selected for publication. +Raw and per-request evidence stays in the ignored `../results` tree or durable +external storage. Every published report must retain its campaign manifest, +normalized metrics, and provenance. Link native evidence only when it has a +durable archive URL; do not commit links into the ignored local results tree. diff --git a/inference/reports/llm-d-benchmark/optimized-baseline-qwen3-32b-h100/optimized-baseline-v0230-gateway-refresh-20260817/README.md b/inference/reports/llm-d-benchmark/optimized-baseline-qwen3-32b-h100/optimized-baseline-v0230-gateway-refresh-20260817/README.md new file mode 100644 index 0000000..e05a439 --- /dev/null +++ b/inference/reports/llm-d-benchmark/optimized-baseline-qwen3-32b-h100/optimized-baseline-v0230-gateway-refresh-20260817/README.md @@ -0,0 +1,8 @@ +# Inference Benchmark Campaign + +Campaign: `optimized-baseline-v0230-gateway-refresh-20260817` + +## Comparisons + +- [service vs agentgateway-standalone](service-vs-agentgateway-standalone/README.md) +- [service vs agentgateway-gateway](service-vs-agentgateway-gateway/README.md) diff --git a/inference/reports/llm-d-benchmark/optimized-baseline-qwen3-32b-h100/optimized-baseline-v0230-gateway-refresh-20260817/campaign-manifest.yaml b/inference/reports/llm-d-benchmark/optimized-baseline-qwen3-32b-h100/optimized-baseline-v0230-gateway-refresh-20260817/campaign-manifest.yaml new file mode 100644 index 0000000..92b9879 --- /dev/null +++ b/inference/reports/llm-d-benchmark/optimized-baseline-qwen3-32b-h100/optimized-baseline-v0230-gateway-refresh-20260817/campaign-manifest.yaml @@ -0,0 +1,135 @@ +schema_version: 1 +campaign_id: optimized-baseline-v0230-gateway-refresh-20260817 +suite: llm-d-benchmark +identity: + scenario: optimized-baseline + cluster_provider: gke + accelerator_type: gpu + accelerator_model: h100 + backend_type: vllm + model: Qwen/Qwen3-32B + replicas: 8 + tensor_parallelism: 2 + routing_policy: optimized-baseline + workload_variant: upstream + harness: inference-perf + workload: upstream-optimized-baseline.yaml + workload_sha256: b2ad162508976d18ed7ef21a8a0f8ed0431c92039e9c9b0ee43a2452882e63f7 + scenario_source_sha256: 4ae7851be32a23cfd2272fe30d40032541261921e0cb18226537fa1cffb8246e + shared_config_sha256: d6a29f6eaa1360c24ad91c9f94bb914d33d28ec0ee6b146b11ba77f88ebcd2b4 + llm_d_benchmark_ref: main + reference_profile: optimized-baseline-qwen3-32b-h100-v0.9 + endpoint_path: internal + storage: + model: + profile: high-throughput-shared + strategy: filestore + class: premium-rwx + size: 2560Gi + workload: + profile: shared + class: standard-rwx + size: 100Gi + runtime_metrics: + enabled: true + interval_seconds: 5 + gke_monitoring: auto + result_collection: + fast_collect: true + gpu_release: + policy: after-load + node_pool: gpu-h100 +treatments: + service: + repetitions: + '1': + config_sha256: df9d8e0b527a6f1b88fdeedf6c0d34a5ba7c925ee69834493b3fefa223e46158 + gateway: + implementation: none + mode: none + image: not-applicable + agentgateway_version: not-applicable + storage: + model: + profile: high-throughput-shared + strategy: filestore + class: premium-rwx + size: 2560Gi + workload: + profile: shared + class: standard-rwx + size: 100Gi + router_chart_version: v0.9.0 + router_chart_digest: not-applicable + standalone_invariant_sha256: not-applicable + runtime_metrics: + enabled: true + interval_seconds: 5 + gke_monitoring: auto + result_collection: + fast_collect: true + gpu_release: + policy: after-load + node_pool: gpu-h100 + agentgateway-standalone: + repetitions: + '1': + config_sha256: a3b689e6b16f29c395bc32784ca4cd24a1f3ea25e061921fc1905ee29250864c + gateway: + implementation: agentgateway + mode: standalone + image: cr.agentgateway.dev/agentgateway:v1.4.1 + agentgateway_version: v1.4.1 + storage: + model: + profile: high-throughput-shared + strategy: filestore + class: premium-rwx + size: 2560Gi + workload: + profile: shared + class: standard-rwx + size: 100Gi + router_chart_version: v0.9.0 + router_chart_digest: sha256:8150c84045d59e6e91991daf810322140e116d5e02d244d96cbfc2b3000825cd + standalone_invariant_sha256: d27aa3658a2688014b3e7d88d84a9e22264b1b9eda9d1b60838bfa740d5886ee + runtime_metrics: + enabled: true + interval_seconds: 5 + gke_monitoring: auto + result_collection: + fast_collect: true + gpu_release: + policy: after-load + node_pool: gpu-h100 + agentgateway-gateway: + repetitions: + '1': + config_sha256: 63ee4f0ab209f078f7c81e60cf94ab6a1bb7bcf47a3632c48e6d9b817c4f0c6c + gateway: + implementation: agentgateway + mode: gateway + image: cr.agentgateway.dev/agentgateway:v1.4.1 + agentgateway_version: v1.4.1 + storage: + model: + profile: high-throughput-shared + strategy: filestore + class: premium-rwx + size: 2560Gi + workload: + profile: shared + class: standard-rwx + size: 100Gi + router_chart_version: v0.9.0 + router_chart_digest: not-applicable + standalone_invariant_sha256: not-applicable + runtime_metrics: + enabled: true + interval_seconds: 5 + gke_monitoring: auto + result_collection: + fast_collect: true + gpu_release: + policy: after-load + node_pool: gpu-h100 diff --git a/inference/reports/llm-d-benchmark/optimized-baseline-qwen3-32b-h100/optimized-baseline-v0230-gateway-refresh-20260817/campaign-provenance.yaml b/inference/reports/llm-d-benchmark/optimized-baseline-qwen3-32b-h100/optimized-baseline-v0230-gateway-refresh-20260817/campaign-provenance.yaml new file mode 100644 index 0000000..41e4576 --- /dev/null +++ b/inference/reports/llm-d-benchmark/optimized-baseline-qwen3-32b-h100/optimized-baseline-v0230-gateway-refresh-20260817/campaign-provenance.yaml @@ -0,0 +1,21 @@ +schema_version: 1 +campaign_id: optimized-baseline-v0230-gateway-refresh-20260817 +execution_model: independent-treatment-runs +treatments: + service: + status: complete + harness_start: "2026-08-17T16:46:26+00:00" + harness_stop: "2026-08-17T18:10:01+00:00" + agentgateway-standalone: + status: complete + harness_start: "2026-08-17T21:10:17+00:00" + harness_stop: "2026-08-17T22:09:12+00:00" + agentgateway-gateway: + status: complete + harness_start: "2026-08-18T14:45:27+00:00" + harness_stop: "2026-08-18T15:45:43+00:00" +shared_identity: + workload_sha256: b2ad162508976d18ed7ef21a8a0f8ed0431c92039e9c9b0ee43a2452882e63f7 + scenario_source_sha256: 4ae7851be32a23cfd2272fe30d40032541261921e0cb18226537fa1cffb8246e + shared_config_sha256: d6a29f6eaa1360c24ad91c9f94bb914d33d28ec0ee6b146b11ba77f88ebcd2b4 + inference_perf_revision: e28d9a0bf5cefa743910b73057b3b686f0a94b98 diff --git a/inference/reports/llm-d-benchmark/optimized-baseline-qwen3-32b-h100/optimized-baseline-v0230-gateway-refresh-20260817/service-vs-agentgateway-gateway/README.md b/inference/reports/llm-d-benchmark/optimized-baseline-qwen3-32b-h100/optimized-baseline-v0230-gateway-refresh-20260817/service-vs-agentgateway-gateway/README.md new file mode 100644 index 0000000..98ab87a --- /dev/null +++ b/inference/reports/llm-d-benchmark/optimized-baseline-qwen3-32b-h100/optimized-baseline-v0230-gateway-refresh-20260817/service-vs-agentgateway-gateway/README.md @@ -0,0 +1,79 @@ +# Benchmark Report + +The benchmark runs model Qwen/Qwen3-32B on 16 × H100 GPUs, distributed across 8 vLLM model servers (2 GPUs per server with TP=2). Workload is `upstream-optimized-baseline.yaml` driven across the configured request-rate ladder. + +> [!NOTE] +> These results use the `optimized-baseline` routing policy. Latency is reported as median (p50) in the summary; NTPOT is end-to-end latency divided by output tokens. Missing latency histograms are shown as `n/a` and dropped from latency lines. Both treatments use the same model-server topology and workload. + +## Comparing agentgateway on Kubernetes to a Simple Kubernetes Service (vLLM) + +**agentgateway mode:** In Kubernetes Gateway API mode, agentgateway is the Gateway data plane; an HTTPRoute targets an InferencePool, whose EPP selects the model-server endpoint. [Learn more about Kubernetes inference routing](https://agentgateway.dev/docs/kubernetes/latest/llm/inference/inference-routing/). + +Graphs below compare agentgateway on Kubernetes routing to a stock Kubernetes Service that round-robins requests across the same 8 vLLM pods (no EPP, no scoring). Tables abbreviate this baseline as `k8s service (RR)`, where RR means round-robin. + +Throughput vs QPS +Latency vs QPS +TTFT p90 vs QPS +Summary (throughput is peak sustained; latencies at the top of the ladder, rate 60): + +| Metric | k8s service (RR) | agentgateway on Kubernetes | Δ% vs k8s | +| :--- | :--- | :--- | :--- | +| Peak output tokens/s | 6,910 | 14,241 | +106.1% | +| Requests/sec (@ rate 60) | 6.70 | 13.96 | +108.3% | +| TTFT p50 (s) | 62.9 | 0.2 | −99.7% | +| TTFT p90 (s) | 135.6 | 0.2 | −99.8% | +| ITL p50 (ms) | 30.3 | 49.6 | +63.7% | + +
+Click to view the per-rate breakdown across the full ladder + +Output tokens/sec — higher is better; TTFT in seconds — lower is better. `n/a` means the stage had no successful-request latency histogram. + +| Rate | k8s service (RR) Output | agentgateway on Kubernetes Output | k8s service (RR) TTFT p50 | agentgateway on Kubernetes TTFT p50 | k8s service (RR) TTFT p90 | agentgateway on Kubernetes TTFT p90 | +| ---: | ---: | ---: | ---: | ---: | ---: | ---: | +| 3 | 1,570 | 1,520 | 0.5 | 0.1 | 0.5 | 0.2 | +| 10 | 5,113 | 4,739 | 0.5 | 0.1 | 1.0 | 0.2 | +| 15 | 4,634 | 6,547 | 0.6 | 0.1 | 1.8 | 0.2 | +| 20 | 6,182 | 10,853 | 2.5 | 0.1 | 36.0 | 0.3 | +| 22 | 6,255 | 11,530 | 3.9 | 0.1 | 36.8 | 0.2 | +| 25 | 6,044 | 11,959 | 6.7 | 0.1 | 41.2 | 0.2 | +| 30 | 6,296 | 12,532 | 7.0 | 0.1 | 42.6 | 0.2 | +| 35 | 6,145 | 12,279 | 7.6 | 0.1 | 45.2 | 0.2 | +| 40 | 6,910 | 14,241 | 78.0 | 0.2 | 124.5 | 0.2 | +| 43 | 6,858 | 14,216 | 80.2 | 0.2 | 129.2 | 0.2 | +| 46 | 6,800 | 14,163 | 65.3 | 0.2 | 132.0 | 0.2 | +| 49 | 6,780 | 14,205 | 55.1 | 0.2 | 131.3 | 0.2 | +| 52 | 6,893 | 14,116 | 72.0 | 0.2 | 133.6 | 0.2 | +| 55 | 6,865 | 14,024 | 56.4 | 0.2 | 133.6 | 0.2 | +| 57 | 6,840 | 13,988 | 55.2 | 0.2 | 134.3 | 0.2 | +| 60 | 6,879 | 13,999 | 62.9 | 0.2 | 135.6 | 0.2 | +
+ +Latency must be interpreted together with failures. A treatment can show deceptively low latency under overload when slower requests time out and are excluded from successful-request latency distributions. Failure counts and rates are retained in `metrics.csv`. + +## Interpreting ITL + +At rate 60, k8s service (RR) has a lower p50 ITL (30.3 ms) than agentgateway on Kubernetes (49.6 ms). ITL measures token cadence only after the first response token; time waiting to enter prefill or decode appears in TTFT instead. Lower ITL therefore does not by itself mean better overall inference latency. + +Here, k8s service (RR) peaks at 6,910 output tokens/s and has 62.9 s TTFT p50, while agentgateway on Kubernetes reaches 14,241 output tokens/s with 0.2 s TTFT p50. This is consistent with the Service queueing longer while the routed treatment keeps more work decoding; vLLM continuous batching can trade higher per-stream ITL for greater throughput and lower TTFT. At 3 QPS, ITL is 16.1 ms versus 15.6 ms, indicating that the larger high-load gap is not a fixed proxy penalty. + +## Configuration + +- Campaign: `optimized-baseline-v0230-gateway-refresh-20260817` +- Suite: `llm-d-benchmark` +- Scenario: `optimized-baseline` +- Model: `Qwen/Qwen3-32B` +- Backend: `vllm` +- Backend image/version: `docker.io/vllm/vllm-openai:v0.23.0` +- Accelerator: `h100` +- Model servers: 8 with TP=2 +- Workload: `upstream-optimized-baseline.yaml` (`upstream`) +- Repetitions per treatment: 1 +- Candidate components: agentgateway cr.agentgateway.dev/agentgateway:v1.4.1 + +Warm-up stages whose requested rate is repeated in the measured ladder are excluded. With multiple repetitions, each table and graph point is the median at that requested QPS. + +## Evidence + +- [Campaign manifest](../campaign-manifest.yaml) +- [Normalized metrics](metrics.csv) diff --git a/inference/reports/llm-d-benchmark/optimized-baseline-qwen3-32b-h100/optimized-baseline-v0230-gateway-refresh-20260817/service-vs-agentgateway-gateway/latency_vs_qps.png b/inference/reports/llm-d-benchmark/optimized-baseline-qwen3-32b-h100/optimized-baseline-v0230-gateway-refresh-20260817/service-vs-agentgateway-gateway/latency_vs_qps.png new file mode 100644 index 0000000..6c00df6 Binary files /dev/null and b/inference/reports/llm-d-benchmark/optimized-baseline-qwen3-32b-h100/optimized-baseline-v0230-gateway-refresh-20260817/service-vs-agentgateway-gateway/latency_vs_qps.png differ diff --git a/inference/reports/llm-d-benchmark/optimized-baseline-qwen3-32b-h100/optimized-baseline-v0230-gateway-refresh-20260817/service-vs-agentgateway-gateway/metrics.csv b/inference/reports/llm-d-benchmark/optimized-baseline-qwen3-32b-h100/optimized-baseline-v0230-gateway-refresh-20260817/service-vs-agentgateway-gateway/metrics.csv new file mode 100644 index 0000000..03dadfb --- /dev/null +++ b/inference/reports/llm-d-benchmark/optimized-baseline-qwen3-32b-h100/optimized-baseline-v0230-gateway-refresh-20260817/service-vs-agentgateway-gateway/metrics.csv @@ -0,0 +1,33 @@ +treatment,requested_qps,input_tokens_per_second,output_tokens_per_second,total_tokens_per_second,requests_per_second,ttft_mean_seconds,ttft_p50_seconds,ttft_p90_seconds,itl_mean_seconds,itl_p50_seconds,ntpot_mean_seconds,ntpot_p50_seconds,failures,requests,failure_rate +service,3.0,11503.596220598163,1569.9745971620982,13073.570817760261,1.5977216973053003,0.44429938492442794,0.4855520714772865,0.5148372672265396,0.01792539998956845,0.01612949196714908,0.018592630535022563,0.01798880284995539,0.0,60.0,0.0 +service,10.0,33930.645310927925,5112.735611705391,39043.38092263332,4.712589626517768,0.5624921136262128,0.5064612334826961,0.9502927532885223,0.02343723281651085,0.022294262889772654,0.032769966581297605,0.02651632518526569,0.0,200.0,0.0 +service,15.0,32320.229472369305,4634.017642085351,36954.24711445466,4.488920760051293,1.0216799809182218,0.5718230929924175,1.7923080221167766,0.031494547249183664,0.02792685345048085,0.07619524823443098,0.03522006891183779,0.0,300.0,0.0 +service,20.0,43775.307432263464,6181.518202398546,49956.82563466201,6.079903810036592,16.788055226358118,2.482193897420075,36.01030341892038,0.04038861760144181,0.030047064006794244,0.1304162470146181,0.057537559968428784,0.0,760.0,0.0 +service,22.0,44084.89004399636,6254.543774991984,50339.43381898835,6.122901394999495,17.791630647738963,3.868383436987642,36.78848744758871,0.04170006155730675,0.0302008839789778,0.13018376439914467,0.06428183773589656,0.0,748.0,0.0 +service,25.0,42769.64790727726,6043.802706692854,48813.45061397012,5.940228876010731,21.96000503822354,6.693649045540951,41.225694391143044,0.041361125439183834,0.0301172339823097,0.15575572449258587,0.06769425680530139,0.0,750.0,0.0 +service,30.0,43794.63239267638,6295.81092124873,50090.44331392511,6.082587832316165,21.665828232227824,6.96267025306588,42.61537396111525,0.04126903953512137,0.030230033909901977,0.1318563050225732,0.0729189679340049,0.0,750.0,0.0 +service,35.0,42853.69858796039,6145.489225358025,48999.187813318415,5.951902581661166,22.438835869022,7.638133830041625,45.19686885757837,0.04138130536121994,0.030223564011976123,0.12231134744660388,0.07254900645944304,0.0,735.0,0.0 +service,40.0,48684.803707217034,6910.212671808169,55595.016379025205,6.761778292669033,64.5031793560361,78.04157234349987,124.53171786206076,0.043335432052543255,0.030391194042749703,0.28793932495281355,0.1266460353852854,0.0,1520.0,0.0 +service,43.0,48727.86102954411,6858.261944166925,55586.12297371103,6.767758476325571,68.08422804851138,80.18435114051681,129.18915484364842,0.0431951379729301,0.03040753398090601,0.25330468792281247,0.12852774474727865,0.0,1548.0,0.0 +service,46.0,47929.87528934324,6800.143148552775,54730.01843789601,6.6569271235198935,68.02844357919827,65.34275427903049,131.96879429563415,0.04333255647592945,0.03043846396030858,0.23217964727087856,0.13058881069748896,0.0,1518.0,0.0 +service,49.0,47477.90542529427,6779.565876807093,54257.47130210137,6.594153531290871,66.92081123598832,55.118164095969405,131.3106305822381,0.043285183941279004,0.030430893995799124,0.2363770809176936,0.13053083742951277,0.0,1470.0,0.0 +service,52.0,48820.20307453384,6892.975982947252,55713.179057481095,6.780583760351923,69.68626782034107,71.96844228345435,133.61431046595098,0.04328036125612368,0.03038660401944071,0.2867025356968608,0.13251005062738574,0.0,1508.0,0.0 +service,55.0,48351.4213258416,6865.137575538113,55216.558901379714,6.715475184144667,68.11642817942392,56.38790225004777,133.57044004225173,0.04355175149221362,0.030485464027151465,0.24170138750332917,0.13196492378297262,0.0,1485.0,0.0 +service,57.0,48021.38318356429,6840.004813300278,54861.38799686457,6.669636553272818,68.92170332540009,55.24773368757451,134.33478256077507,0.04313486986530813,0.030462573980912566,0.2294224301939611,0.1321347232350381,0.0,1482.0,0.0 +service,60.0,48253.72950880769,6878.537865915115,55132.26737472281,6.701906876223291,70.50749036829197,62.939835166034754,135.59904143075693,0.043198107088589614,0.030317444005049765,0.2205583167749312,0.1338126110171024,0.0,1500.0,0.0 +agentgateway-gateway,3.0,11735.218855796444,1519.8195012594895,13255.038357055933,1.6298915077495062,0.09755827240102614,0.08437465253518894,0.153943062084727,0.016048950964576952,0.015583681990392506,0.040150530423425444,0.015821306832014194,0.0,60.0,0.0 +agentgateway-gateway,10.0,36498.970878404216,4738.504240796381,41237.47511920059,5.069301510889474,0.1051851022260962,0.09108537354040891,0.16457930088508874,0.02114766138387135,0.02066288294736296,0.04660460908411748,0.020408251288537654,0.0,200.0,0.0 +agentgateway-gateway,15.0,50514.68290546541,6546.656131693775,57061.33903715918,7.01592818131464,0.11061932488693856,0.09673328447388485,0.17632086697267368,0.023191031602105054,0.02210458304034546,0.046809383712322836,0.02259789268017154,0.0,300.0,0.0 +agentgateway-gateway,20.0,78659.76247947564,10852.545979720377,89512.30845919602,10.924967011038285,0.1634470776556746,0.13021234900224954,0.27101435597287493,0.03011305212672444,0.02858168398961425,0.07321859176986191,0.032739041396279464,0.0,760.0,0.0 +agentgateway-gateway,22.0,82526.96828099198,11530.207808815969,94057.17608980794,11.46207892791555,0.1251839015876904,0.1216544879716821,0.1647082926821895,0.028148118065259266,0.028283444000408053,0.05928768200610499,0.029535697526657315,0.0,748.0,0.0 +agentgateway-gateway,25.0,85515.28482572008,11958.869170288968,97474.15399600904,11.877122892461122,0.12715814948299278,0.12162108748452738,0.16450151301687585,0.029112974877806914,0.029184124490711838,0.06995047008779551,0.030581532662790248,0.0,750.0,0.0 +agentgateway-gateway,30.0,89313.75932008504,12532.225534848327,101845.98485493337,12.404688794456257,0.1297818652872617,0.12506367301102728,0.16809358585160225,0.030611081241000457,0.03130654490087181,0.0611811813441837,0.03229806135453474,0.0,750.0,0.0 +agentgateway-gateway,35.0,88222.78604742691,12279.404839398607,100502.19088682551,12.253164728809292,0.12713131305356162,0.12272466800641268,0.1617549579590559,0.03183723844472639,0.03338963503483683,0.0679218473402897,0.03295879343303386,0.0,735.0,0.0 +agentgateway-gateway,40.0,102648.16951983512,14241.214198688016,116889.38371852314,14.25669021108821,0.8401657825250847,0.15800215798662975,0.22505092100473123,0.045845804884798065,0.04477624595165253,0.10028814170674194,0.04878695761447307,0.0,1520.0,0.0 +agentgateway-gateway,43.0,101587.6026997147,14216.203173915832,115803.80587363052,14.109389263849263,0.6040474720024299,0.15980589797254652,0.22805287500377744,0.048636114361967,0.046435027034021914,0.10603016549156905,0.0508677457542728,0.0,1548.0,0.0 +agentgateway-gateway,46.0,101239.43005266342,14163.284739450155,115402.71479211358,14.061031951758808,0.501574799470982,0.16070223896531388,0.23567670931806786,0.04863243305786868,0.04617465706542134,0.10701641535379455,0.05053191233185661,0.0,1518.0,0.0 +agentgateway-gateway,49.0,101322.29484158699,14204.96643250912,115527.2612740961,14.072540950220414,0.3728833490924048,0.15811813302570954,0.22942429806571454,0.048046386955449924,0.045671446539927274,0.09765176696472688,0.049688919483138416,0.0,1470.0,0.0 +agentgateway-gateway,52.0,100705.87495331428,14115.91659598564,114821.79154929993,13.986927076849206,0.16329726050612492,0.15902379801264033,0.23160648198099804,0.05119188085979826,0.04815255245193839,0.10557478004699389,0.05289131667218407,0.0,1508.0,0.0 +agentgateway-gateway,55.0,99525.62626675333,14023.665257347651,113549.29152410098,13.823003648160183,0.16419200921409233,0.1578219230286777,0.2316369260195643,0.051117387533717876,0.04769907705485821,0.1058993245597146,0.05231983479152514,0.0,1485.0,0.0 +agentgateway-gateway,57.0,99612.71780606132,13988.471390896186,113601.18919695751,13.835099695286296,0.16724619867401724,0.1624154740711674,0.23524426094954834,0.0521097812807918,0.04836323799099773,0.11086446715585457,0.053139202505222374,0.0,1482.0,0.0 +agentgateway-gateway,60.0,100525.5348159039,13999.390752700365,114524.92556860426,13.961879835542208,0.16787564251517567,0.16036469850223511,0.24196021403186024,0.05314929263347045,0.0496409279294312,0.11624340388269203,0.05385783021081909,0.0,1500.0,0.0 diff --git a/inference/reports/llm-d-benchmark/optimized-baseline-qwen3-32b-h100/optimized-baseline-v0230-gateway-refresh-20260817/service-vs-agentgateway-gateway/throughput_vs_qps.png b/inference/reports/llm-d-benchmark/optimized-baseline-qwen3-32b-h100/optimized-baseline-v0230-gateway-refresh-20260817/service-vs-agentgateway-gateway/throughput_vs_qps.png new file mode 100644 index 0000000..cb8464d Binary files /dev/null and b/inference/reports/llm-d-benchmark/optimized-baseline-qwen3-32b-h100/optimized-baseline-v0230-gateway-refresh-20260817/service-vs-agentgateway-gateway/throughput_vs_qps.png differ diff --git a/inference/reports/llm-d-benchmark/optimized-baseline-qwen3-32b-h100/optimized-baseline-v0230-gateway-refresh-20260817/service-vs-agentgateway-gateway/ttft_p90_vs_qps.png b/inference/reports/llm-d-benchmark/optimized-baseline-qwen3-32b-h100/optimized-baseline-v0230-gateway-refresh-20260817/service-vs-agentgateway-gateway/ttft_p90_vs_qps.png new file mode 100644 index 0000000..7f5a31d Binary files /dev/null and b/inference/reports/llm-d-benchmark/optimized-baseline-qwen3-32b-h100/optimized-baseline-v0230-gateway-refresh-20260817/service-vs-agentgateway-gateway/ttft_p90_vs_qps.png differ diff --git a/inference/reports/llm-d-benchmark/optimized-baseline-qwen3-32b-h100/optimized-baseline-v0230-gateway-refresh-20260817/service-vs-agentgateway-standalone/README.md b/inference/reports/llm-d-benchmark/optimized-baseline-qwen3-32b-h100/optimized-baseline-v0230-gateway-refresh-20260817/service-vs-agentgateway-standalone/README.md new file mode 100644 index 0000000..19a9816 --- /dev/null +++ b/inference/reports/llm-d-benchmark/optimized-baseline-qwen3-32b-h100/optimized-baseline-v0230-gateway-refresh-20260817/service-vs-agentgateway-standalone/README.md @@ -0,0 +1,79 @@ +# Benchmark Report + +The benchmark runs model Qwen/Qwen3-32B on 16 × H100 GPUs, distributed across 8 vLLM model servers (2 GPUs per server with TP=2). Workload is `upstream-optimized-baseline.yaml` driven across the configured request-rate ladder. + +> [!NOTE] +> These results use the `optimized-baseline` routing policy. Latency is reported as median (p50) in the summary; NTPOT is end-to-end latency divided by output tokens. Missing latency histograms are shown as `n/a` and dropped from latency lines. Both treatments use the same model-server topology and workload. + +## Comparing agentgateway standalone to a Simple Kubernetes Service (vLLM) + +**agentgateway mode:** In standalone request scheduler mode, agentgateway runs as a sidecar proxy with the EPP and communicates with it over localhost, without a full Gateway API stack. [Learn more about standalone inference routing](https://agentgateway.dev/docs/standalone/latest/inference/#standalone-request-scheduler-mode). + +Graphs below compare agentgateway standalone routing to a stock Kubernetes Service that round-robins requests across the same 8 vLLM pods (no EPP, no scoring). Tables abbreviate this baseline as `k8s service (RR)`, where RR means round-robin. + +Throughput vs QPS +Latency vs QPS +TTFT p90 vs QPS +Summary (throughput is peak sustained; latencies at the top of the ladder, rate 60): + +| Metric | k8s service (RR) | agentgateway standalone | Δ% vs k8s | +| :--- | :--- | :--- | :--- | +| Peak output tokens/s | 6,910 | 16,178 | +134.1% | +| Requests/sec (@ rate 60) | 6.70 | 16.52 | +146.5% | +| TTFT p50 (s) | 62.9 | 0.1 | −99.8% | +| TTFT p90 (s) | 135.6 | 0.2 | −99.8% | +| ITL p50 (ms) | 30.3 | 52.7 | +73.7% | + +
+Click to view the per-rate breakdown across the full ladder + +Output tokens/sec — higher is better; TTFT in seconds — lower is better. `n/a` means the stage had no successful-request latency histogram. + +| Rate | k8s service (RR) Output | agentgateway standalone Output | k8s service (RR) TTFT p50 | agentgateway standalone TTFT p50 | k8s service (RR) TTFT p90 | agentgateway standalone TTFT p90 | +| ---: | ---: | ---: | ---: | ---: | ---: | ---: | +| 3 | 1,570 | 1,694 | 0.5 | 0.1 | 0.5 | 0.1 | +| 10 | 5,113 | 4,723 | 0.5 | 0.1 | 1.0 | 0.2 | +| 15 | 4,634 | 6,480 | 0.6 | 0.1 | 1.8 | 0.2 | +| 20 | 6,182 | 10,974 | 2.5 | 0.2 | 36.0 | 0.3 | +| 22 | 6,255 | 11,831 | 3.9 | 0.1 | 36.8 | 0.1 | +| 25 | 6,044 | 12,227 | 6.7 | 0.1 | 41.2 | 0.1 | +| 30 | 6,296 | 12,923 | 7.0 | 0.1 | 42.6 | 0.1 | +| 35 | 6,145 | 13,032 | 7.6 | 0.1 | 45.2 | 0.1 | +| 40 | 6,910 | 15,964 | 78.0 | 0.1 | 124.5 | 0.2 | +| 43 | 6,858 | 15,781 | 80.2 | 0.1 | 129.2 | 0.2 | +| 46 | 6,800 | 15,566 | 65.3 | 0.2 | 132.0 | 0.2 | +| 49 | 6,780 | 15,834 | 55.1 | 0.1 | 131.3 | 0.2 | +| 52 | 6,893 | 16,031 | 72.0 | 0.1 | 133.6 | 0.2 | +| 55 | 6,865 | 15,766 | 56.4 | 0.1 | 133.6 | 0.2 | +| 57 | 6,840 | 15,829 | 55.2 | 0.1 | 134.3 | 0.2 | +| 60 | 6,879 | 16,178 | 62.9 | 0.1 | 135.6 | 0.2 | +
+ +Latency must be interpreted together with failures. A treatment can show deceptively low latency under overload when slower requests time out and are excluded from successful-request latency distributions. Failure counts and rates are retained in `metrics.csv`. + +## Interpreting ITL + +At rate 60, k8s service (RR) has a lower p50 ITL (30.3 ms) than agentgateway standalone (52.7 ms). ITL measures token cadence only after the first response token; time waiting to enter prefill or decode appears in TTFT instead. Lower ITL therefore does not by itself mean better overall inference latency. + +Here, k8s service (RR) peaks at 6,910 output tokens/s and has 62.9 s TTFT p50, while agentgateway standalone reaches 16,178 output tokens/s with 0.1 s TTFT p50. This is consistent with the Service queueing longer while the routed treatment keeps more work decoding; vLLM continuous batching can trade higher per-stream ITL for greater throughput and lower TTFT. At 3 QPS, ITL is 16.1 ms versus 15.7 ms, indicating that the larger high-load gap is not a fixed proxy penalty. + +## Configuration + +- Campaign: `optimized-baseline-v0230-gateway-refresh-20260817` +- Suite: `llm-d-benchmark` +- Scenario: `optimized-baseline` +- Model: `Qwen/Qwen3-32B` +- Backend: `vllm` +- Backend image/version: `docker.io/vllm/vllm-openai:v0.23.0` +- Accelerator: `h100` +- Model servers: 8 with TP=2 +- Workload: `upstream-optimized-baseline.yaml` (`upstream`) +- Repetitions per treatment: 1 +- Candidate components: agentgateway cr.agentgateway.dev/agentgateway:v1.4.1 + +Warm-up stages whose requested rate is repeated in the measured ladder are excluded. With multiple repetitions, each table and graph point is the median at that requested QPS. + +## Evidence + +- [Campaign manifest](../campaign-manifest.yaml) +- [Normalized metrics](metrics.csv) diff --git a/inference/reports/llm-d-benchmark/optimized-baseline-qwen3-32b-h100/optimized-baseline-v0230-gateway-refresh-20260817/service-vs-agentgateway-standalone/latency_vs_qps.png b/inference/reports/llm-d-benchmark/optimized-baseline-qwen3-32b-h100/optimized-baseline-v0230-gateway-refresh-20260817/service-vs-agentgateway-standalone/latency_vs_qps.png new file mode 100644 index 0000000..26ec349 Binary files /dev/null and b/inference/reports/llm-d-benchmark/optimized-baseline-qwen3-32b-h100/optimized-baseline-v0230-gateway-refresh-20260817/service-vs-agentgateway-standalone/latency_vs_qps.png differ diff --git a/inference/reports/llm-d-benchmark/optimized-baseline-qwen3-32b-h100/optimized-baseline-v0230-gateway-refresh-20260817/service-vs-agentgateway-standalone/metrics.csv b/inference/reports/llm-d-benchmark/optimized-baseline-qwen3-32b-h100/optimized-baseline-v0230-gateway-refresh-20260817/service-vs-agentgateway-standalone/metrics.csv new file mode 100644 index 0000000..b894112 --- /dev/null +++ b/inference/reports/llm-d-benchmark/optimized-baseline-qwen3-32b-h100/optimized-baseline-v0230-gateway-refresh-20260817/service-vs-agentgateway-standalone/metrics.csv @@ -0,0 +1,33 @@ +treatment,requested_qps,input_tokens_per_second,output_tokens_per_second,total_tokens_per_second,requests_per_second,ttft_mean_seconds,ttft_p50_seconds,ttft_p90_seconds,itl_mean_seconds,itl_p50_seconds,ntpot_mean_seconds,ntpot_p50_seconds,failures,requests,failure_rate +service,3.0,11503.596220598163,1569.9745971620982,13073.570817760261,1.5977216973053003,0.44429938492442794,0.4855520714772865,0.5148372672265396,0.01792539998956845,0.01612949196714908,0.018592630535022563,0.01798880284995539,0.0,60.0,0.0 +service,10.0,33930.645310927925,5112.735611705391,39043.38092263332,4.712589626517768,0.5624921136262128,0.5064612334826961,0.9502927532885223,0.02343723281651085,0.022294262889772654,0.032769966581297605,0.02651632518526569,0.0,200.0,0.0 +service,15.0,32320.229472369305,4634.017642085351,36954.24711445466,4.488920760051293,1.0216799809182218,0.5718230929924175,1.7923080221167766,0.031494547249183664,0.02792685345048085,0.07619524823443098,0.03522006891183779,0.0,300.0,0.0 +service,20.0,43775.307432263464,6181.518202398546,49956.82563466201,6.079903810036592,16.788055226358118,2.482193897420075,36.01030341892038,0.04038861760144181,0.030047064006794244,0.1304162470146181,0.057537559968428784,0.0,760.0,0.0 +service,22.0,44084.89004399636,6254.543774991984,50339.43381898835,6.122901394999495,17.791630647738963,3.868383436987642,36.78848744758871,0.04170006155730675,0.0302008839789778,0.13018376439914467,0.06428183773589656,0.0,748.0,0.0 +service,25.0,42769.64790727726,6043.802706692854,48813.45061397012,5.940228876010731,21.96000503822354,6.693649045540951,41.225694391143044,0.041361125439183834,0.0301172339823097,0.15575572449258587,0.06769425680530139,0.0,750.0,0.0 +service,30.0,43794.63239267638,6295.81092124873,50090.44331392511,6.082587832316165,21.665828232227824,6.96267025306588,42.61537396111525,0.04126903953512137,0.030230033909901977,0.1318563050225732,0.0729189679340049,0.0,750.0,0.0 +service,35.0,42853.69858796039,6145.489225358025,48999.187813318415,5.951902581661166,22.438835869022,7.638133830041625,45.19686885757837,0.04138130536121994,0.030223564011976123,0.12231134744660388,0.07254900645944304,0.0,735.0,0.0 +service,40.0,48684.803707217034,6910.212671808169,55595.016379025205,6.761778292669033,64.5031793560361,78.04157234349987,124.53171786206076,0.043335432052543255,0.030391194042749703,0.28793932495281355,0.1266460353852854,0.0,1520.0,0.0 +service,43.0,48727.86102954411,6858.261944166925,55586.12297371103,6.767758476325571,68.08422804851138,80.18435114051681,129.18915484364842,0.0431951379729301,0.03040753398090601,0.25330468792281247,0.12852774474727865,0.0,1548.0,0.0 +service,46.0,47929.87528934324,6800.143148552775,54730.01843789601,6.6569271235198935,68.02844357919827,65.34275427903049,131.96879429563415,0.04333255647592945,0.03043846396030858,0.23217964727087856,0.13058881069748896,0.0,1518.0,0.0 +service,49.0,47477.90542529427,6779.565876807093,54257.47130210137,6.594153531290871,66.92081123598832,55.118164095969405,131.3106305822381,0.043285183941279004,0.030430893995799124,0.2363770809176936,0.13053083742951277,0.0,1470.0,0.0 +service,52.0,48820.20307453384,6892.975982947252,55713.179057481095,6.780583760351923,69.68626782034107,71.96844228345435,133.61431046595098,0.04328036125612368,0.03038660401944071,0.2867025356968608,0.13251005062738574,0.0,1508.0,0.0 +service,55.0,48351.4213258416,6865.137575538113,55216.558901379714,6.715475184144667,68.11642817942392,56.38790225004777,133.57044004225173,0.04355175149221362,0.030485464027151465,0.24170138750332917,0.13196492378297262,0.0,1485.0,0.0 +service,57.0,48021.38318356429,6840.004813300278,54861.38799686457,6.669636553272818,68.92170332540009,55.24773368757451,134.33478256077507,0.04313486986530813,0.030462573980912566,0.2294224301939611,0.1321347232350381,0.0,1482.0,0.0 +service,60.0,48253.72950880769,6878.537865915115,55132.26737472281,6.701906876223291,70.50749036829197,62.939835166034754,135.59904143075693,0.043198107088589614,0.030317444005049765,0.2205583167749312,0.1338126110171024,0.0,1500.0,0.0 +agentgateway-standalone,3.0,12510.137544349538,1693.9073738869952,14204.044918236534,1.7375191033818802,0.09271785082528368,0.08236233098432422,0.14574138699099423,0.015971692592934635,0.01574079308193177,0.023732712632805852,0.015966119172894568,0.0,60.0,0.0 +agentgateway-standalone,10.0,36017.63185092261,4722.962049918273,40740.59390084088,5.002448868183696,0.1011386097164359,0.09074482799042016,0.15758009703131393,0.020444174324226086,0.020270462962798774,0.03795532060812598,0.02026308712549508,0.0,200.0,0.0 +agentgateway-standalone,15.0,49726.359562432335,6479.551844149776,56205.91140658211,6.9064388281156015,0.1082823058058663,0.09225567255634815,0.17433033816050741,0.0234130324538457,0.022458893014118075,0.06185693182404866,0.023184566911027514,0.0,300.0,0.0 +agentgateway-standalone,20.0,81559.08720861122,10973.989813095504,92533.07702170673,11.327651001196003,0.16723987929596518,0.18089464004151523,0.26251649095211177,0.03221262062824708,0.029749304056167603,0.07003282627759161,0.034012317798507864,0.0,760.0,0.0 +agentgateway-standalone,22.0,86870.92776123577,11831.131277963488,98702.05903919926,12.065406633504967,0.11296319188413155,0.11328274552943185,0.14295276999473572,0.029235889848613906,0.029075884027406573,0.06053174250234363,0.0306601055426553,0.0,748.0,0.0 +agentgateway-standalone,25.0,89871.2245094591,12227.163165524802,102098.38767498391,12.482114515202653,0.11436559714920198,0.11348456551786512,0.14458988894475625,0.029891549613425775,0.030257283942773938,0.0651932983855052,0.03154649156500818,0.0,750.0,0.0 +agentgateway-standalone,30.0,95020.05091081475,12923.413179794052,107943.4640906088,13.197229293168716,0.11418153046645844,0.11496204050490633,0.14500819389941172,0.031355496830390545,0.0332676749676466,0.0661934209947222,0.032736343990025236,0.0,750.0,0.0 +agentgateway-standalone,35.0,95443.29175907168,13032.22960672168,108475.52136579336,13.256012744315512,0.11396611973700323,0.11405018495861441,0.14507259542588144,0.031718199180180005,0.03358738502720371,0.05863366013493191,0.03290967886183258,0.0,735.0,0.0 +agentgateway-standalone,40.0,116390.42870021569,15963.557750420197,132353.9864506359,16.1653373194744,0.15218902263915976,0.14955064054811373,0.216460024996195,0.04734050594202779,0.04811155691277236,0.08880179982606586,0.04947220601745245,0.0,1520.0,0.0 +agentgateway-standalone,43.0,117277.14821400202,15781.31880664659,133058.46702064862,16.28849280750028,0.15170186774119931,0.14851725549669936,0.2105408629751764,0.04847514175970665,0.04911227198317647,0.12089965448665682,0.050318506902491206,0.0,1548.0,0.0 +agentgateway-standalone,46.0,115179.70131943954,15565.646776970749,130745.3480964103,15.997180738811048,0.1603710834133716,0.15004719607532024,0.215410582954064,0.04904106754386426,0.050097651954274625,0.13144208190966075,0.05068815032368404,0.0,1518.0,0.0 +agentgateway-standalone,49.0,116622.02678473621,15834.161757401318,132456.18854213753,16.19750372010225,0.15052274533263946,0.14610010100295767,0.21260604993440213,0.04958095402712243,0.05063390696886927,0.12711260292247287,0.05150110965192454,0.0,1470.0,0.0 +agentgateway-standalone,52.0,116467.41163774466,16031.174552660152,132498.5861904048,16.176029394131206,0.15162366571604888,0.14889842603588477,0.21325146600138395,0.051668012486085726,0.051930287561845034,0.10467026304254118,0.053973419188857744,0.0,1508.0,0.0 +agentgateway-standalone,55.0,116453.30000970105,15766.167038509424,132219.46704821047,16.17406944579181,0.15551058237632165,0.14985053101554513,0.220731963054277,0.05174320947990799,0.051735347020439804,0.10553004037040908,0.05414679543499369,0.0,1485.0,0.0 +agentgateway-standalone,57.0,116198.14979817063,15829.242799423926,132027.39259759456,16.13863191641259,0.15024550031168654,0.14556700049433857,0.21336815601680428,0.05136456224604826,0.05155014700721949,0.11053361066106505,0.05339643597982375,0.0,1482.0,0.0 +agentgateway-standalone,60.0,118951.28673607696,16178.234088732894,135129.52082480985,16.521012046677356,0.1519910846018077,0.14820081100333482,0.21448402515379714,0.05319834085155357,0.05267447791993618,0.12110073974948303,0.054958149435347214,0.0,1500.0,0.0 diff --git a/inference/reports/llm-d-benchmark/optimized-baseline-qwen3-32b-h100/optimized-baseline-v0230-gateway-refresh-20260817/service-vs-agentgateway-standalone/throughput_vs_qps.png b/inference/reports/llm-d-benchmark/optimized-baseline-qwen3-32b-h100/optimized-baseline-v0230-gateway-refresh-20260817/service-vs-agentgateway-standalone/throughput_vs_qps.png new file mode 100644 index 0000000..6089292 Binary files /dev/null and b/inference/reports/llm-d-benchmark/optimized-baseline-qwen3-32b-h100/optimized-baseline-v0230-gateway-refresh-20260817/service-vs-agentgateway-standalone/throughput_vs_qps.png differ diff --git a/inference/reports/llm-d-benchmark/optimized-baseline-qwen3-32b-h100/optimized-baseline-v0230-gateway-refresh-20260817/service-vs-agentgateway-standalone/ttft_p90_vs_qps.png b/inference/reports/llm-d-benchmark/optimized-baseline-qwen3-32b-h100/optimized-baseline-v0230-gateway-refresh-20260817/service-vs-agentgateway-standalone/ttft_p90_vs_qps.png new file mode 100644 index 0000000..9e6467d Binary files /dev/null and b/inference/reports/llm-d-benchmark/optimized-baseline-qwen3-32b-h100/optimized-baseline-v0230-gateway-refresh-20260817/service-vs-agentgateway-standalone/ttft_p90_vs_qps.png differ