vLLM-offload-bench

Benchmarks of CPU KV-cache tiers for vLLM on one node: vLLM's built-in OffloadingConnector, LMCacheMPConnector (LMCache multiprocess server, CUDA IPC) and MooncakeStoreConnector (Mooncake store, TCP), against plain vLLM.

Everything here is reproducible with one script per experiment; the numbers in results/ and report.html were produced on 8× RTX A6000 (PCIe Gen4 x16, no RDMA), Llama-3.1-8B-Instruct bf16.

Install: docs/INSTALL.md (uv, pinned vLLM/LMCache commits + patches). Full write-up with all tables and pitfalls: RESULTS.md. Single-page report: report.html.

Headline (GPU prefix caching off, one instance per setup, repeat of a ~31k-token prompt)

	plain vLLM	Mooncake store	LMCache MP	vLLM native
repeat TTFT (KV from CPU)	6.8 s	0.76 s	0.43 s	0.35 s
first-request cost vs plain	1.00×	1.5–2.2×	1.05–1.07×	1.00×
4 × 8k concurrent repeats, wall	5.2 s	0.9 / 2.6 s (bimodal)	0.62 s	0.53 s
8 × 17.5k concurrent repeats, wall	27 s	3.5 s	2.25 s	1.37 s
shares one pool across engines	—	yes (TCP, ~0.8 GB/s cross-process here)	yes (CUDA IPC, ~9 GB/s)	no

Why native is faster on a hit (from per-request GPU timelines, plots/timeline_*.png): one cuMemcpyBatchAsync from pinned host straight into the paged KV at PCIe line rate (139 ms for 3.7 GB) vs LMCache's two-stage staging (173 ms); no cross-process RPC before/after the copy (~18 + ~10 ms); 16-token instead of 256-token hit granularity (fewer tail tokens recomputed). Both saturate the PCIe link while copying (26–27 GB/s peak); all three use cudaHostRegister-pinned host memory.

Layout

bench/        run_all.sh (launch 4 setups → sweeps → e2e → PCIe probe → batch×length sweep → report → teardown)
              bench_offload.py  sweep_conc.py  e2e_gen.py  pcie_probe.py   (measurements)
              aggregate.py  plot.py  plot_sweep.py  build_report.py          (tables / figures / HTML)
              nsys_capture.sh  nsys_timeline.py  nsys_loads.py               (per-request GPU timelines)
results/      run_all_2pass/ (canonical)  sweep_batch_x_length/  timelines/  hand_run_passes/  validation_1pass_3setups/
plots/        all figures (PNG)           report.html   self-contained report with figures inlined
patches/      local vLLM / LMCache diffs the numbers were taken with
requirements-bench.txt

Run

source .venv/bin/activate                       # see docs/INSTALL.md
bench/run_all.sh                                # ~25 min: 4 setups, 2 alternating passes, e2e, report
PCIE_PROBE=1 bench/run_all.sh                   # + NVML PCIe sampling during store/load
SWEEP=1 PASSES=0 bench/run_all.sh               # batch-size × sequence-length sweep (~30 min)
GPUS=4,5,6,7 CPU_TIER_GB=40 bench/run_all.sh    # every knob is an env var (top of run_all.sh)
bench/nsys_capture.sh results/my_timelines      # per-request timelines (torch profiler + nsys)

Each run writes results/run_<timestamp>/ with config.env, raw per-request JSON, summary.md, plots/, report.html and every server log. Re-plot or rebuild the report from any results dir:

python bench/aggregate.py --results results/run_all_2pass
python bench/plot.py      --results results/run_all_2pass --out plots
python bench/plot_sweep.py --sweep results/sweep_batch_x_length/sweep.json --out plots
python bench/build_report.py --results results/run_all_2pass --plots plots \
    --sweep results/sweep_batch_x_length/sweep.json --timelines results/timelines --out report.html