Skip to content

Add vLLM hosting and client integrations - #2535

Open
luisquintanilla wants to merge 9 commits into
CommunityToolkit:mainfrom
luisquintanilla:luisquintanilla-vllm-integration
Open

luisquintanilla wants to merge 9 commits into
CommunityToolkit:mainfrom
luisquintanilla:luisquintanilla-vllm-integration

Conversation

@luisquintanilla

@luisquintanilla luisquintanilla commented Nov 16, 2026 •

Copy link
Copy Markdown

Closes #1484

This adds an Aspire hosting integration for running vLLM and a client integration for consuming its OpenAI-compatible API. The APIs follow the existing Ollama/OllamaSharp builder pattern while supporting GPU configuration, Hugging Face model setup, and Microsoft.Extensions.AI IChatClient registration.

In the AppHost:

var vllm = builder.AddVLLM("vllm")
    .WithGPUSupport() // NVIDIA by default; pass VLLMGpuVendor.AMD for ROCm
    .WithDataVolume()
    .WithModel("Qwen/Qwen3-0.6B");

builder.AddProject<Projects.Api>("api")
    .WithReference(vllm)
    .WaitFor(vllm);

In the consuming app:

builder.AddVLLMClient("vllm")
    .AddChatClient("Qwen/Qwen3-0.6B");

The client appends /v1 to the hosting endpoint. The client model ID must match the name served by vLLM; Hugging Face credentials are only needed for gated models and should be supplied as a secret parameter.

PR Checklist

  • Created a feature/dev branch in your fork (vs. submitting directly from a commit on main)
  • Based off latest main branch of toolkit
  • PR doesn't include merge commits (always rebase on top of our main, if needed)
  • New integration
    • Docs are written - package READMEs are included; the full aspire.dev guide is a separate follow-up, per CONTRIBUTING.md.
    • Added description of major feature to project description for NuGet package (4000 total character limit, so don't push entire description over that)
  • Tests for the changes have been added (for bug fixes / features) (if applicable)
  • Contains NO breaking changes
  • Every new API (including internal ones) has full XML docs
  • Code follows all style conventions

Other information

Suggested review order

  1. Hosting: src/CommunityToolkit.Aspire.Hosting.VLLM — resource/connection properties, image defaults, shared-memory sizing, and NVIDIA/AMD/Podman runtime arguments.
  2. Client: src/CommunityToolkit.Aspire.VLLM — settings precedence, default/keyed registration, model resolution, health checks, and telemetry opt-outs.
  3. Tests and examples: hosting/client tests, tests-app-hosts/VLLM.AppHost, and C#/TypeScript/Go examples; finish with solution, CODEOWNERS, and README integration-table registration.

API shape for review: VLLM naming, the integration-specific VLLMGpuVendor, and the fluent AddVLLMClient(name).AddChatClient() shape. Hosting and client are combined because they implement the same issue and share the consumer example. Issue #1484 is open and assigned; I found no explicit maintainer approval there, so this PR does not claim formal approval.

Review follow-up: Removed the fabricated token from the public-model Go example, made repeated WithModel calls replace the model, and gave distinct keyed clients independent health checks. The corresponding review threads have been answered and resolved.

Validation

Local checks used the repository-pinned .NET SDK 11.0.100 and matching Aspire CLI. NuGet audit was disabled because the local vulnerability-data endpoint was unavailable; npm dependencies were restored from a locally warmed offline cache.

Validation Result
dotnet build src\CommunityToolkit.Aspire.Hosting.VLLM\CommunityToolkit.Aspire.Hosting.VLLM.csproj --no-restore --configuration Release --verbosity quiet Passed for net10.0 and net11.0, no warnings.
dotnet build src\CommunityToolkit.Aspire.VLLM\CommunityToolkit.Aspire.VLLM.csproj --no-restore --configuration Release --verbosity quiet Passed for net10.0 and net11.0, no warnings.
dotnet build tests\CommunityToolkit.Aspire.Hosting.VLLM.Tests\CommunityToolkit.Aspire.Hosting.VLLM.Tests.csproj --no-restore --configuration Release --verbosity quiet Passed; two existing ASPIRE012 warnings from staged AppHost SDK references.
dotnet test --project tests\CommunityToolkit.Aspire.Hosting.VLLM.Tests\CommunityToolkit.Aspire.Hosting.VLLM.Tests.csproj --no-restore --hangdump --hangdump-timeout 7m --verbosity quiet 19 passed, 0 skipped in 2m 06s, including both real CPU Docker AppHosts. The C# test waits for health and checks /v1/models; the TypeScript test waits for health.
dotnet test --project tests\CommunityToolkit.Aspire.VLLM.Tests\CommunityToolkit.Aspire.VLLM.Tests.csproj --no-restore --verbosity quiet 23 passed, 0 skipped, including independent keyed health-check probes and collision cases.
go test ./... in the Go AppHost example Passed compilation; no Go test files. The hosting suite also restores and validates the Go AppHost model.
git diff --check and outgoing credential/path scan Passed.

CPU startup: The C# smoke host and TypeScript CPU example use --enforce-eager, without changing integration defaults or limiting the sequence count. This disables graph compilation and JIT warmup for these low-resource workloads, potentially reducing inference throughput. With a warm model cache, two containers sharing four CPUs and limited to 8 GiB each reached health and listed models in 70.1s and 82.6s, then both served completions; PyTorch/oneDNN were constrained to AVX2. In the same local setup with one sequence, compiled execution took 267.2s and 299.3s with JIT warmup enabled, versus 147.4s and 168.0s with only JIT warmup disabled; both latter servers also served completions. This implicates JIT warmup as a significant startup cost, although the exact blocked worker frame in the earlier hosted timeout is not known.

Hosted CI: The .NET CI workflow passed for commit 3c27e13c; the Ubuntu hosting suite passed all 19 tests with no skips in 2m 48s, including both CPU Docker hosts. Windows hosting and client tests, Ubuntu client tests, both builds, and CodeQL also passed. The remaining integration-table check fails with NETSDK1005, the repository-level issue tracked by open PR #2530.

Limits and follow-ups: GPU/ROCm/Podman behavior is covered by app-model assertions, not a hardware/runtime test of this port. The C# CPU Docker smoke test checks readiness and model listing; the separate constrained experiment also exercised a completion. Go and TypeScript validate generated Aspire AppHosts over the same hosting integration; they are not alternate vLLM implementations. Full aspire.dev documentation belongs in a separate PR to microsoft/aspire.dev, as described in CONTRIBUTING.md. No full-repository test run is claimed.

luisquintanilla and others added 4 commits November 16, 2026 12:04
Refs CommunityToolkit#1484

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Refs CommunityToolkit#1484

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Refs CommunityToolkit#1484

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Document internal integration members and inherited API contracts for the PR checklist. No runtime behavior changes.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 16, 2026 17:59
@github-actions

Copy link
Copy Markdown
Contributor

🚀 Dogfood this PR with:

⚠️ WARNING: Do not do this without first carefully reviewing the code of this PR to satisfy yourself it is safe.

curl -fsSL https://raw.githubusercontent.com/CommunityToolkit/Aspire/main/eng/scripts/dogfood-pr.sh | bash -s -- 2535

Or

  • Run remotely in PowerShell:
iex "& { $(irm https://raw.githubusercontent.com/CommunityToolkit/Aspire/main/eng/scripts/dogfood-pr.ps1) } 2535"

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unresolved issues cause authentication failures in the example, invalid startup arguments when overriding models, and missing health checks for keyed clients with distinct endpoints.

3 open issues
1 file not reviewed
  • examples/vllm/CommunityToolkit.Aspire.Hosting.VLLM.AppHost.TypeScript/package-lock.json: Generated file
What changed in this PR

Adds paired vLLM hosting and client integrations for #1484, extending the toolkit’s existing model-serving support with an OpenAI-compatible inference server.

Changes:

  • Adds container configuration for GPUs, model selection, Hugging Face caching and tokens, and readiness checks.
  • Adds default and keyed IChatClient registration with configuration binding, /v1 routing, health checks, and optional telemetry.
  • Adds C#, TypeScript, and Go examples, focused tests, package documentation, and repository registration.
File Description
tests/​CommunityToolkit.Aspire.VLLM.Tests/​VLLMClientSettingsTests.cs Tests connection-string parsing.
tests/​CommunityToolkit.Aspire.VLLM.Tests/​CommunityToolkit.Aspire.VLLM.Tests.csproj Defines the client test project.
tests/​CommunityToolkit.Aspire.VLLM.Tests/​AddVLLMClientTests.cs Tests client configuration and registration.
tests/​CommunityToolkit.Aspire.Hosting.VLLM.Tests/​TypeScriptAppHostTests.cs Tests TypeScript AppHost startup.
tests/​CommunityToolkit.Aspire.Hosting.VLLM.Tests/​GoAppHostTests.cs Validates the Go AppHost.
tests/​CommunityToolkit.Aspire.Hosting.VLLM.Tests/​CommunityToolkit.Aspire.Hosting.VLLM.Tests.csproj Wires hosting test dependencies.
tests/​CommunityToolkit.Aspire.Hosting.VLLM.Tests/​AppHostTests.cs Checks readiness and model listing.
tests/​CommunityToolkit.Aspire.Hosting.VLLM.Tests/​AddVLLMTests.cs Tests hosting resource configuration.
tests-app-hosts/​VLLM.AppHost/​VLLM.AppHost.csproj Defines the smoke-test AppHost.
tests-app-hosts/​VLLM.AppHost/​AppHost.cs Configures a small CPU model for testing.
src/​CommunityToolkit.Aspire.VLLM/​VLLMHealthCheck.cs Probes server readiness.
src/​CommunityToolkit.Aspire.VLLM/​VLLMClientSettings.cs Defines connection and telemetry settings.
src/​CommunityToolkit.Aspire.VLLM/​README.md Documents client setup and configuration.
src/​CommunityToolkit.Aspire.VLLM/​CommunityToolkit.Aspire.VLLM.csproj Defines the client package and dependencies.
src/​CommunityToolkit.Aspire.VLLM/​AspireVLLMExtensions.cs Registers client settings and health checks.
src/​CommunityToolkit.Aspire.VLLM/​AspireVLLMClientBuilder.cs Carries fluent registration state.
src/​CommunityToolkit.Aspire.VLLM/​AspireVLLMChatClientExtensions.cs Creates chat clients and telemetry wrappers.
src/​CommunityToolkit.Aspire.Hosting.VLLM/​VLLMResourceBuilderExtensions.cs Adds container and model configuration APIs.
src/​CommunityToolkit.Aspire.Hosting.VLLM/​VLLMResource.cs Exposes endpoints and connection properties.
src/​CommunityToolkit.Aspire.Hosting.VLLM/​VLLMGpuVendor.cs Defines GPU vendor options.
src/​CommunityToolkit.Aspire.Hosting.VLLM/​VLLMContainerImageTags.cs Centralizes default container images.
src/​CommunityToolkit.Aspire.Hosting.VLLM/​README.md Documents hosting and hardware setup.
src/​CommunityToolkit.Aspire.Hosting.VLLM/​CommunityToolkit.Aspire.Hosting.VLLM.csproj Defines the hosting package.
README.md Lists both integrations.
examples/​vllm/​CommunityToolkit.Aspire.Hosting.VLLM.ConsumerApp/​Worker.cs Demonstrates a chat request.
examples/​vllm/​CommunityToolkit.Aspire.Hosting.VLLM.ConsumerApp/​Program.cs Registers the example chat client.
examples/​vllm/​CommunityToolkit.Aspire.Hosting.VLLM.ConsumerApp/​CommunityToolkit.Aspire.Hosting.VLLM.ConsumerApp.csproj Wires consumer dependencies.
examples/​vllm/​CommunityToolkit.Aspire.Hosting.VLLM.AppHost/​Properties/​launchSettings.json Defines local launch profiles.
examples/​vllm/​CommunityToolkit.Aspire.Hosting.VLLM.AppHost/​CommunityToolkit.Aspire.Hosting.VLLM.AppHost.csproj Connects hosting and consumer projects.
examples/​vllm/​CommunityToolkit.Aspire.Hosting.VLLM.AppHost/​appsettings.json Configures example logging.
examples/​vllm/​CommunityToolkit.Aspire.Hosting.VLLM.AppHost/​AppHost.cs Connects the GPU server and consumer.
examples/​vllm/​CommunityToolkit.Aspire.Hosting.VLLM.AppHost.TypeScript/​tsconfig.json Configures TypeScript compilation.
examples/​vllm/​CommunityToolkit.Aspire.Hosting.VLLM.AppHost.TypeScript/​package.json Defines TypeScript tooling and scripts.
examples/​vllm/​CommunityToolkit.Aspire.Hosting.VLLM.AppHost.TypeScript/​package-lock.json Locks JavaScript dependencies.
examples/​vllm/​CommunityToolkit.Aspire.Hosting.VLLM.AppHost.TypeScript/​eslint.config.mjs Checks asynchronous AppHost calls.
examples/​vllm/​CommunityToolkit.Aspire.Hosting.VLLM.AppHost.TypeScript/​aspire.config.json Configures the SDK and hosting reference.
examples/​vllm/​CommunityToolkit.Aspire.Hosting.VLLM.AppHost.TypeScript/​apphost.mts Demonstrates CPU hosting in TypeScript.
examples/​vllm/​CommunityToolkit.Aspire.Hosting.VLLM.AppHost.Go/​go.mod References generated Go modules.
examples/​vllm/​CommunityToolkit.Aspire.Hosting.VLLM.AppHost.Go/​aspire.config.json Configures the Go AppHost SDK.
examples/​vllm/​CommunityToolkit.Aspire.Hosting.VLLM.AppHost.Go/​apphost.go Demonstrates hosting APIs in Go.
CommunityToolkit.Aspire.slnx Registers packages, examples, and tests.
CODEOWNERS Assigns ownership of the integration.

Review effort: Balanced

luisquintanilla and others added 5 commits November 16, 2026 14:35
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

vLLM hosting and client integration

2 participants