Compare agentgateway v1.6.0 and Praxis release and nightly builds

.github/workflows/ai-gateway-tooling.yaml

@@ -0,0 +1,54 @@
+name: AI gateway tooling
+on:
+  pull_request:
+    paths: ['ai-gateway/**', 'cpu-comparison/**', '.github/workflows/ai-gateway-tooling.yaml']
+  push:
+    branches: [main]
+    paths: ['ai-gateway/**', 'cpu-comparison/**', '.github/workflows/ai-gateway-tooling.yaml']
+permissions:
+  contents: read
+jobs:
+  verify:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
+      - uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
+        with:
+          python-version: '3.12'
+      - name: Validate harness
+        run: |
+          bash -n ai-gateway/build-mock.sh
+          shellcheck ai-gateway/build-mock.sh
+          python3 -m unittest discover -s ai-gateway -p 'test_*.py'
+          python3 ai-gateway/campaign.py --initial --dry-run > /tmp/ai-plan.json
+          GOTOOLCHAIN=go1.27.1 CGO_ENABLED=0 go build -trimpath -o /tmp/mock-server ai-gateway/mock/main.go
+      - name: Validate retained CPU comparison snapshots
+        run: |
+          python3 - <<'PYCODE'
+          import ast
+          import json
+          import hashlib
+          import subprocess
+          from pathlib import Path
+          root = Path('cpu-comparison')
+          for path in root.rglob('*.py'):
+              ast.parse(path.read_text(), filename=str(path))
+          for path in root.rglob('*.sh'):
+              subprocess.run(['bash', '-n', str(path)], check=True)
+          for path in root.rglob('*.json'):
+              json.loads(path.read_text())
+          current = root / '2026-10-05-stable-comparison'
+          data = current / 'reports/data'
+          rows = json.loads((data / 'load-rows.json').read_text())
+          groups = json.loads((data / 'load-summary.json').read_text())
+          review = json.loads((current / 'evidence/qualification/final-review.json').read_text())
+          assert review['infrastructure_valid'] and review['all_three_passes_reviewed']
+          claims = json.loads((data / 'article-calculations.json').read_text())
+          assert claims['source_sha256'] == hashlib.sha256((data / 'load-summary.json').read_bytes()).hexdigest()
+          assert len(rows) == 822 and len(groups) == 274
+          assert all(g['passes'] == [1, 2, 3] and g['three_complete_passes'] for g in groups)
+          assert all(r['artifact_valid'] and all(v == 0 for v in r['exits']) for r in rows)
+          assert len(json.loads((data / 'resource-rows.json').read_text())) == 606
+          assert len(json.loads((data / 'resource-summary.json').read_text())) == 202
+          print('Frozen snapshot syntax, JSON and complete three-pass dimensions verified')
+          PYCODE

.gitignore

@@ -3,3 +3,10 @@ results/
 __pycache__/
 *.pyc
 .venv/
+ai-gateway/mock/mock-server
+!ai-gateway/results/
+ai-gateway/results/*
+!ai-gateway/results/2026-10-01-initial/
+!gateway-api/results/
+gateway-api/results/*
+!gateway-api/results/2026-10-01-initial/

README.md

@@ -3,20 +3,49 @@
 Community benchmarks for [agentgateway](https://github.com/agentgateway/agentgateway).
 
 This repo holds the tooling and published results for measuring how agentgateway
-performs as an inference gateway - both as EPP's standalone sidecar and as a
-Gateway API data plane - compared to routing traffic through a plain Kubernetes
-Service with no gateway at all.
+performs as an AI proxy, Kubernetes Gateway API implementation and inference gateway. CPU-only AI proxy tests use a
+deterministic backend; inference tests compare EPP integration against a plain
+Kubernetes Service. These workloads answer different questions.
+
+## Current CPU comparison: agentgateway v1.6.0 and Praxis
+
+The [three-pass comparison](cpu-comparison/2026-10-05-stable-comparison/README.md)
+separates agentgateway versus direct service from Praxis versus agentgateway.
+It includes core v0.5.2 and core nightly-20261002 for transport, plus the separate
+Praxis AI v0.5.0 and October 2 AI nightly for native AI workloads. It runs on
+native GCE VMs with dedicated client/gateway/backend roles, using Fortio,
+Nighthawk, AIPerf and ClusterLoader2. Streaming uses a deterministic CPU service;
+these are not GPU inference or model-quality results.
+
+The [extended Gateway API evidence](cpu-comparison/2026-10-02-gateway-extended/README.md)
+retains every selected upstream case and the original suite version. The newer
+campaign adds a separate corrected static-address check and lifecycle evidence.
+Read full matrices, exclusions, version pins and source-supported feature
+boundaries together; no overall gateway score is implied.
+
+Earlier October 1 results below are historical, single-run experiments with
+agentgateway v1.5.0 and different infrastructure. Do not relabel them as v1.6.0
+or Praxis nightly measurements.
 
 ## Layout
 
+- [`cpu-comparison/`](cpu-comparison/2026-10-05-stable-comparison/README.md) - current
+  three-pass CPU comparison and retained extended Gateway API evidence.
+
+- [`ai-gateway/`](ai-gateway/README.md) - native CPU-only protocol qualification
+  and Fortio proxy measurements for agentgateway, Praxis AI, and a direct backend.
+
+- [`gateway-api/`](gateway-api/README.md) - shared core HTTP conformance, plain
+  HTTP traffic, route lifecycle and scale/churn evidence for agentgateway and Praxis.
+
 - [`inference/`](inference/README.md) - the benchmark runner itself: campaign-based
   execution comparing `service`, `agentgateway-standalone`, and
   `agentgateway-gateway` treatments, with automated GKE provisioning/teardown and
   Markdown/PNG/CSV report generation.
 - [`inference/reports/`](inference/reports/README.md) - curated, published benchmark
   results with their campaign manifests and provenance, safe to link to directly.
 
-## Quick start
+## Inference quick start
 
 Run the `service` treatment locally on Kind:
 
@@ -33,9 +62,15 @@ than most laptops provide for the router pod - see
 See [`inference/README.md`](inference/README.md) for the full set of treatments,
 GKE campaign instructions, and configuration reference.
 
-## Current results
+## Historical CPU and other inference results
+
+The [initial CPU AI comparison](ai-gateway/results/2026-10-01-initial/README.md)
+and [Gateway API comparison](gateway-api/results/2026-10-01-initial/README.md)
+retain direct baselines, separate paired reports and full raw evidence. These
+are single-round experiments with workload-specific outcomes and disclosed
+failures, not GPU inference results.
 
-The most recent published campaign
+The previously published inference campaign
 ([`optimized-baseline-v0230-gateway-refresh-20260817`](inference/reports/llm-d-benchmark/optimized-baseline-qwen3-32b-h100/optimized-baseline-v0230-gateway-refresh-20260817/README.md))
 ran Qwen/Qwen3-32B on 16x H100 GPUs across 8 vLLM model servers. At the top of the
 request-rate ladder, agentgateway more than doubles peak throughput and cuts TTFT

ai-gateway/README.md

@@ -0,0 +1,132 @@
+# AI gateway protocol and proxy-overhead campaign
+
+This is the historical October 1, single-run campaign. For the completed
+three-pass agentgateway v1.6.0 and Praxis release/nightly comparison, start with
+[the current CPU reports](../cpu-comparison/2026-10-05-stable-comparison/README.md).
+
+This harness uses a deterministic Go backend, Fortio 1.75.1, agentgateway v1.5.0
+and Praxis AI 0.5.0, with container images pinned in `compose.yaml`. It measures
+synthetic proxy overhead; it performs **no model inference**. The [initial single-round report](results/2026-10-01-initial/README.md) includes
+all 54 trials and the complete raw evidence archive.
+
+The workload design follows the public
+[agentgateway/LiteLLM proxy benchmark](https://github.com/linsun/litellm-agw-perf)
+and its fixed-throughput follow-up. The implementation here adds repeatable
+payloads, a direct backend, explicit resource isolation, alternating treatment
+order, protocol qualification and retained failure evidence. It is newly written;
+no source from that harness was copied.
+
+## What is compared
+
+| Case | Client API | Backend API | Enabled processing |
+| --- | --- | --- | --- |
+| `openai` | Chat Completions | Chat Completions | agentgateway AI provider route; Praxis token usage extraction + routing |
+| `anthropic` | Messages | Messages | agentgateway Anthropic provider; Praxis format/protocol filters, token extraction + routing |
+| `translation` | Messages | Chat Completions | agentgateway provider translation; Praxis Messages-to-Chat request/response and SSE translation |
+
+Configurations use synthetic credentials. No external authorization, distributed
+rate limiting, response cache, guardrail service, TLS, database or real provider
+is enabled. Agentgateway uses two workers; Praxis sets `runtime.threads: 2` (per service).
+One API listener receives traffic per trial. Both have the same two logical CPUs and
+2 GiB memory limit. The mock and load generator have separate two-CPU sets and
+2 GiB limits. Record physical core/SMT layout; disjoint logical CPU IDs alone do
+not ensure independent physical cores. Access logging is disabled for
+agentgateway; Praxis has no access-log filter. Their built-in AI processing is
+not identical, so this is a comparison of the documented configurations, not a
+proof of equal internal work.
+
+The direct baseline uses the backend wire format. For translation it sends Chat
+Completions directly, whereas clients send Messages to the gateways. Use it to
+check backend headroom; subtracting its latency does not isolate translation
+cost precisely.
+
+## Run
+
+Requirements: Python 3.10+, Go 1.26+, Docker with Compose, and a dedicated native
+Linux/amd64 Docker host with at least eight CPUs. The benchmark project name and
+loopback ports 28080–28084 are fixed; the runner refuses to reuse existing project
+containers. Do not run on a shared host for publication.
+
+```sh
+cd ai-gateway
+python3 campaign.py --smoke --dry-run
+python3 campaign.py --smoke --output /tmp/ai-qualification-001
+python3 campaign.py --initial --dry-run
+python3 campaign.py --initial --output /tmp/ai-initial-001
+# For a separately planned repeated campaign:
+python3 campaign.py --output /tmp/ai-performance-001
+```
+
+The runner rebuilds the mock for the Docker host architecture, creates only its
+Compose services, and removes those services after success or failure. It
+refuses to overwrite a campaign. `build-mock.sh` is also available separately.
+The default CPU sets are gateway `2-3`, mock `4-5`, client `6-7`; override
+`GATEWAY_CPUSET`, `MOCK_CPUSET`, and `CLIENT_CPUSET` after inspecting topology.
+
+Full defaults: three cases, 1,024/16,384 input-content and output-content bytes,
+1,000/3,000 offered QPS and maximum throughput, 32 connections, five-second
+warmup, 30-second measurement, six repetitions. JSON framing adds bytes beyond
+the content size. The 324 measured trials require at least 189 minutes of
+warmup/load, plus startup and collection. Each treatment occupies each position
+twice per workload. A smaller declared subset can be chosen with `--cases`,
+`--sizes`, and `--qps`; preserve the preregistered subset and all outcomes.
+
+`--initial` selects exactly one preliminary round (54 trials, at least 31.5
+minutes of warmup/load for the full matrix). It preserves the 30-second minimum
+and native/resource checks. This explicitly authorized screening mode cannot
+estimate variance or support statistical superiority claims.
+
+Repeated performance mode requires at least five repetitions and 30 seconds, native
+amd64 images and nonoverlapping CPU sets. ARM/emulation is allowed only with
+`--smoke`, whose short load is qualification-only. The earlier local Praxis image crashed under QEMU; native amd64 protocol
+qualification passed. Emulation failures are not native-platform performance
+results.
+
+## Qualification and evidence
+
+Before load, `probe.py` checks JSON content/usage, incremental SSE content and
+terminal events, and an upstream 429 for each selected case. It records first
+semantic output and completion timing at low load; those timings are diagnostic,
+not streaming throughput results. The stream emits byte chunks at a fixed
+cadence. Synthetic usage fields are constants, not tokenizer measurements.
+
+The campaign stores source/binary hashes, image inspections, resolved Compose,
+commands, protocol outcomes, Fortio JSON/logs, resource samples, per-trial
+summaries, cleanup status and SHA-256 checksums. Failures remain incomplete and
+retain their logs. A clean orchestration exit does not prove that offered load
+was sustained; inspect `target_met`, successes, errors and achieved rate.
+
+Fortio uses fixed concurrency and paced traffic with catch-up disabled. Under
+load it can fall below offered QPS; its completion latency histogram is not a
+corrected open-loop latency distribution. Report offered and successful QPS,
+HTTP errors and timeouts alongside percentiles, and use inference-perf for
+inference arrivals, TTFT, token latency and SLO goodput. HTTP 200 validation
+happens in qualification, not on every Fortio response; add sampled protocol
+validation if errors appear only under concurrency.
+
+Docker resource samples are approximate, including startup/stop boundaries;
+filter by the exact trial's container names. The named load generator is
+included so a client bottleneck is detectable. Sampled maxima are not true
+instantaneous peaks, and Docker's memory display is not process RSS. For public
+CPU/RSS claims, collect cgroup/process measurements with timestamps on the native
+host and retain raw data. Do not select the best repetition or equate a
+mock-backend win to improved GPU efficiency.
+
+## Initial reports
+
+Generate the two Markdown comparisons from all trials of a completed initial
+campaign. The report command rejects smoke timings, incomplete campaigns and
+missing treatment pairs:
+
+```sh
+python3 report.py --campaign /tmp/ai-initial-001 --output /tmp/ai-reports \
+  --evidence-link PATH_OR_URL_TO_RAW_EVIDENCE
+python3 -m pip install -r plot-requirements.txt
+python3 plots.py --campaign /tmp/ai-initial-001 --output /tmp/ai-reports
+```
+
+The paired reports use direct backend → agentgateway and agentgateway → Praxis.
+The saturation ratio always means candidate/baseline. Retain raw error counts
+and achieved rates, including cases where either gateway loses. These scripts
+support the initial one-round report; repeated campaigns need a separate
+analysis of dispersion and treatment order.

ai-gateway/build-mock.sh

@@ -0,0 +1,9 @@
+#!/usr/bin/env bash
+set -euo pipefail
+cd "$(dirname "$0")"
+case "$(docker info --format '{{.Architecture}}')" in
+  x86_64|amd64) mock_arch=amd64 ;;
+  aarch64|arm64) mock_arch=arm64 ;;
+  *) echo 'Unsupported Docker architecture' >&2; exit 1 ;;
+esac
+GOTOOLCHAIN=local CGO_ENABLED=0 GOOS=linux GOARCH="$mock_arch" go build -trimpath -o mock/mock-server mock/main.go