VibeServe: Can AI Agents Build Bespoke LLM Serving Systems?

An agentic loop that synthesizes bespoke LLM serving systems — one per (model, hardware, workload) target — instead of forcing every deployment through a single general-purpose runtime.

Updates

• 2026-05 — Blog post: Let AI Agents Write Your Serving Stack with VibeServe.
• 2026-05 — Paper released on arXiv: 2605.06068.

Introduction

VibeServe explores a new approach to LLM serving: instead of relying on one general-purpose runtime to support every model, workload, and hardware target, we use AI agents to generate bespoke serving systems for each deployment scenario. The project asks whether long-horizon coding agents can synthesize complete LLM serving stacks end-to-end, including scheduling, caching, runtime logic, correctness checks, and performance optimizations tailored to a specific target.

The system is organized as a multi-agent optimization loop. An outer loop plans the search over system designs using persistent state such as issues, memory, and git history, while an inner loop implements candidate systems, validates correctness against a reference implementation, and measures performance on the target benchmark. Across standard and non-standard serving scenarios, VibeServe matches highly optimized systems like vLLM in mainstream deployments and achieves substantial gains in specialized settings involving predicted-output decoding, hybrid prompt caching, streaming ASR, constrained JSON decoding, multimodal inference, and Apple Silicon deployment.

Architecture

The framework factors the work along two axes:

• Outer loop — a search policy operating over a git-recorded history of validated checkpoints. It picks the next optimization, dispatches one concrete task to the inner loop, and updates persistent planning state (issues, long-term memory file, commit graph).
• Inner loop — three role-specialized coding-agent invocations on a shared workspace:
  • Implementer writes/edits the candidate serving system.
  • Accuracy Judge runs the user-supplied checker against the reference and inspects diffs/runtime behavior for reward-hacking patterns; only correct candidates exit the inner loop.
  • Performance Evaluator profiles the implementation (Nsight Systems, PyTorch profiler) and feeds bottleneck hints back to the policy.
• Skills library — Agent Skills entries distilled from existing serving engines and research literature (continuous batching, paged-KV, FlashInfer/FlashAttention, MLX, hybrid-cache management, …). New model families, hardware platforms, and optimization techniques are added by writing a skill, not by modifying the framework.
• Execution environment — an isolated workspace that mounts the user-provided artifacts read-only (so the Implementer cannot edit the checker or reference) and exposes the target hardware (local CUDA, Modal, Docker, or Apple Silicon) plus profilers.

Each candidate is a git commit; the outer loop only advances on Judge-validated implementations, so incorrect candidates can never derail subsequent rounds.

Installation

Requires Python 3.11+.

uv sync
cp .env.example .env       # provider keys (Anthropic / OpenAI / Vertex / …)
cp agent.toml.example agent.toml

Quickstart

# Issue-tracker outer loop, Codex CLI, Docker on local CUDA, 4 rounds
vibe-serve \
  --input examples/model-serving/moonshine-streaming \
  --exp-name my-experiment \
  --docker \
  --agent-backend cli --cli-provider codex \
  --max-rounds 4 \
  --modality speech_to_text

--outer-loop defaults to agent. Pass --outer-loop plain or --outer-loop evolve to switch. See vibe-serve --outer-loop <kind> --help for loop-specific flags, and docs/cli-flags.md for the supported flag combinations.

A separate entry point exposes the issue MCP server used by the plain loop:

vibe-serve-issue-mcp                         # serves issues.json over MCP

Domains — pointing vibeserve at your problem space

A domain is the bundle of cross-cutting context the agents need for whatever you're building: the background knowledge the implementer must read, and the correctness/performance/integrity gates the judge enforces. It answers "what kind of system is this, and what does 'good' mean here?" — kept separate from the neutral prompt skeleton and from the per-task I/O contract (--modality).

Select one with --domain (agent loop):

vibe-serve --outer-loop agent --domain llm-serving ...      # default
vibe-serve --outer-loop agent --domain generic ...          # no domain context

--domain takes a registered domain name such as llm-serving or generic. A domain is an in-repo package: prompt templates plus optional environment setup/teardown hooks. The package's templates/ directory contains role files such as implementer.md, judge.md, and optionally single_agent.md; those files drop into the prompts at labelled points. Omit single_agent.md and it's derived from implementer.md plus judge.md.

Full authoring guide: src/vibe_serve/domains/README.md.

Interface — how the artifact is evaluated (and which language)

See the full flag-composition guide in docs/cli-flags.md.

The implementation language is not a user choice. It falls out of --interface, which sets the contract by which the accuracy checker and benchmark reach the built artifact:

vibe-serve --outer-loop agent --interface inprocess ...   # default: in-process Python
vibe-serve --outer-loop agent --interface service ...     # over-the-wire, any language

• inprocess (default): the accuracy checker imports main.py directly, so the implementation is Python. The prompts carry the uv toolchain and the VibeServeModel import contract — vibeserve's original behaviour, unchanged.
• service: the artifact is exercised only through its network interface, so the agent picks whatever language and toolchain fit. The in-process contract and the Python/torch tooling drop out of the prompts; supply an accuracy checker and benchmark that probe the running service over the wire (e.g. HTTP, or the YCSB harness for a key-value store).

Per-target inputs

Each evaluation target lives under examples/<name>/:

examples/<name>/
├── OBJECTIVE.md          # free-form deployment goal (model + hardware + workload + interface)
├── reference/            # reference HuggingFace Transformers implementation
│   ├── reference.py
│   ├── config.json
│   └── meta.json         # model id + revision
├── accuracy_checker/     # checker.py + tests/data — the correctness gate
├── benchmark/            # benchmark.py + load levels — emits the metric to optimize
└── README.md             # human-readable description

Pass the bundle root with --input examples/<name>/ to derive --ref, --acc-checker, and --bench from the standard subdirectory names. You can still pass those three flags explicitly; explicit paths override --input defaults when you need a custom checker or benchmark.

OBJECTIVE.md is read at the start of every run and must live next to the reference/ directory (sibling, not inside). See examples/model-serving/Llama-3-8B/, examples/model-serving/moonshine-streaming/, examples/model-serving/qwen3-32b-code-edit/, examples/model-serving/olmo-hybrid-prefix-caching/, examples/model-serving/Llama-3.1-8B-Instruct-MLX-8bit/, examples/model-serving/show-o2-1.5B-HQ-h100/, and examples/model-serving/show-o2-1.5B-HQ-macbook/ for the paper scenarios.

For multi-objective evolutionary runs, drop an objectives.toml next to OBJECTIVE.md (or pass --objective name:max|min flags) — see vibe-serve --outer-loop evolve --help.

Configuration (agent.toml)

[model]
name = "gpt-5.4"             # auto-detected provider for claude-* / gpt-* / gemini-*
# provider = "openai"        # optional override

[backend]
name = "cuda"                 # or "metal" for Apple Silicon (local exec only)

[agent]
backend = "cli"               # "cli" (codex/claude/gemini/opencode) or "deepagents"
cli_provider = "codex"        # which coding-agent harness to drive
# cli_timeout = 1800          # per-invocation timeout (seconds)

# Optional: benchmark load levels handed to the perf evaluator.
# [[perf_eval.load_levels]]
# rate = 1
# duration = 20
# max_tokens = 128

Provider credentials live in .env — see .env.example. The CLI flags --agent-backend / --cli-provider / --backend override these.

The config is validated against a typed schema on load (vibe_serve/config.py): unknown sections or keys, unknown providers/backends, and missing required fields are rejected with an error rather than silently ignored.

Skills library

resources/skills/ contains the Agent Skills entries the inner loop's agents read at runtime: model architectures, serving algorithms, programming frameworks, backend libraries, hardware platforms, and reference engines. New optimization techniques and model families enter as new skill entries; the framework itself is target-agnostic.

Outputs

Every run creates exp_env/<timestamp>-<name>/:

exp_env/<run>/
├── workspace/                # the unified, git-tracked workspace (each round = one commit)
├── logs/
│   ├── run-*.log             # top-level run log
│   ├── run-*-roundNNN.log    # per-round agent log (agent loop)
│   ├── progress.md           # long-term memory file the Orchestrator reads/edits
│   ├── rounds.json           # per-round audit
│   ├── state.json            # cursor (plain loop)
│   ├── issues.json           # IssueBoard (plain loop)
│   ├── population.json       # Individual list (evolve loop)
│   └── docker.log
└── reference/                # snapshot of --ref at start

Resume any run with --resume (defaults to "latest"):

vibe-serve --resume                  # newest run
vibe-serve --resume 20260507-...     # specific dir

Repository layout

src/vibe_serve/
├── cli.py                        # single entry point: `vibe-serve`
├── context.py                    # _RunContext: lifecycle + ctx.invoke()
├── agent_runner.py               # invoke wrappers + structured-response extraction
├── prompts.py                    # Jinja + backend-fragment renderer
├── schemas.py                    # Pydantic response schemas
├── llm_client.py                 # LLM client factory
├── config.py / constants.py
│
├── loops/                        # the three outer-loop search policies
│   ├── agent/                    # issue-tracker (Orchestrator-driven)
│   ├── plain/                    # Ralph-style queue-drain
│   ├── evolve/                   # population-based
│   └── profiler.py               # shared Performance Evaluator helper
│
├── sandbox/                      # execution-environment policy
│   ├── docker_sandbox.py
│   ├── modal_sandbox.py
│   ├── modal_model_setup.py
│   └── run_environment.py
│
├── agents/                       # coding-agent harness abstraction
│   └── callbacks.py              # LangChain logger (deepagents path)
└── backends/                     # cuda / metal compute backends

examples/                         # six paper scenarios + nsys/torch profiler skills
resources/skills/                 # Agent Skills library

• agent: pre-round → profiler → orchestrator plan → implementer/judge retry up to --max-retries-per-round (default 3). Always exhausts --max-rounds; supports revert_to_round mid-loop.
• plain: drain IssueBoard (one impl + one judge per issue, BLOCK after --max-attempts-per-issue) → perf_eval (may file new issues). Early-exits when queue is empty and perf_eval files nothing.
• evolve: per generation × child: select parent (Pareto frontier with --frontier-bias, scalar softmax otherwise) + inspirations → git checkout parent tree → mutator → judge → profiler → commit. No early stop; runs the full --max-generations × --children-per-generation.

Development

./scripts/format.sh                                # format checked Python dirs
./scripts/check_format.sh                          # check formatting for CI
./scripts/check_lint.sh                            # check Ruff lint for CI
uv run pytest                                       # full suite
uv run pytest tests/loops/plain/test_plain_loop.py  # one file
uv run pytest -k orchestrator                       # by keyword

Citation

If you use VibeServe in your research, please cite:

@misc{kamahori2026vibeserveaiagentsbuild,
      title={VibeServe: Can AI Agents Build Bespoke LLM Serving Systems?},
      author={Keisuke Kamahori and Shihang Li and Simon Peter and Baris Kasikci},
      year={2026},
      eprint={2605.06068},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2605.06068},
}