Repository navigation
Add vLLM hosting and client integrations - #2535
Open
luisquintanilla wants to merge 9 commits into
Open
luisquintanilla wants to merge 9 commits into
luisquintanilla wants to merge 9 commits into
Conversation
Refs CommunityToolkit#1484 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Refs CommunityToolkit#1484 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Refs CommunityToolkit#1484 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Document internal integration members and inherited API contracts for the PR checklist. No runtime behavior changes. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Contributor
|
🚀 Dogfood this PR with:
curl -fsSL https://raw.githubusercontent.com/CommunityToolkit/Aspire/main/eng/scripts/dogfood-pr.sh | bash -s -- 2535Or
iex "& { $(irm https://raw.githubusercontent.com/CommunityToolkit/Aspire/main/eng/scripts/dogfood-pr.ps1) } 2535" |
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved issues cause authentication failures in the example, invalid startup arguments when overriding models, and missing health checks for keyed clients with distinct endpoints.
3 open issues
1 file not reviewed
examples/vllm/CommunityToolkit.Aspire.Hosting.VLLM.AppHost.TypeScript/package-lock.json: Generated file
What changed in this PR
Adds paired vLLM hosting and client integrations for #1484, extending the toolkit’s existing model-serving support with an OpenAI-compatible inference server.
Changes:
- Adds container configuration for GPUs, model selection, Hugging Face caching and tokens, and readiness checks.
- Adds default and keyed
IChatClientregistration with configuration binding,/v1routing, health checks, and optional telemetry. - Adds C#, TypeScript, and Go examples, focused tests, package documentation, and repository registration.
| File | Description |
|---|---|
| tests/CommunityToolkit.Aspire.VLLM.Tests/VLLMClientSettingsTests.cs | Tests connection-string parsing. |
| tests/CommunityToolkit.Aspire.VLLM.Tests/CommunityToolkit.Aspire.VLLM.Tests.csproj | Defines the client test project. |
| tests/CommunityToolkit.Aspire.VLLM.Tests/AddVLLMClientTests.cs | Tests client configuration and registration. |
| tests/CommunityToolkit.Aspire.Hosting.VLLM.Tests/TypeScriptAppHostTests.cs | Tests TypeScript AppHost startup. |
| tests/CommunityToolkit.Aspire.Hosting.VLLM.Tests/GoAppHostTests.cs | Validates the Go AppHost. |
| tests/CommunityToolkit.Aspire.Hosting.VLLM.Tests/CommunityToolkit.Aspire.Hosting.VLLM.Tests.csproj | Wires hosting test dependencies. |
| tests/CommunityToolkit.Aspire.Hosting.VLLM.Tests/AppHostTests.cs | Checks readiness and model listing. |
| tests/CommunityToolkit.Aspire.Hosting.VLLM.Tests/AddVLLMTests.cs | Tests hosting resource configuration. |
| tests-app-hosts/VLLM.AppHost/VLLM.AppHost.csproj | Defines the smoke-test AppHost. |
| tests-app-hosts/VLLM.AppHost/AppHost.cs | Configures a small CPU model for testing. |
| src/CommunityToolkit.Aspire.VLLM/VLLMHealthCheck.cs | Probes server readiness. |
| src/CommunityToolkit.Aspire.VLLM/VLLMClientSettings.cs | Defines connection and telemetry settings. |
| src/CommunityToolkit.Aspire.VLLM/README.md | Documents client setup and configuration. |
| src/CommunityToolkit.Aspire.VLLM/CommunityToolkit.Aspire.VLLM.csproj | Defines the client package and dependencies. |
| src/CommunityToolkit.Aspire.VLLM/AspireVLLMExtensions.cs | Registers client settings and health checks. |
| src/CommunityToolkit.Aspire.VLLM/AspireVLLMClientBuilder.cs | Carries fluent registration state. |
| src/CommunityToolkit.Aspire.VLLM/AspireVLLMChatClientExtensions.cs | Creates chat clients and telemetry wrappers. |
| src/CommunityToolkit.Aspire.Hosting.VLLM/VLLMResourceBuilderExtensions.cs | Adds container and model configuration APIs. |
| src/CommunityToolkit.Aspire.Hosting.VLLM/VLLMResource.cs | Exposes endpoints and connection properties. |
| src/CommunityToolkit.Aspire.Hosting.VLLM/VLLMGpuVendor.cs | Defines GPU vendor options. |
| src/CommunityToolkit.Aspire.Hosting.VLLM/VLLMContainerImageTags.cs | Centralizes default container images. |
| src/CommunityToolkit.Aspire.Hosting.VLLM/README.md | Documents hosting and hardware setup. |
| src/CommunityToolkit.Aspire.Hosting.VLLM/CommunityToolkit.Aspire.Hosting.VLLM.csproj | Defines the hosting package. |
| README.md | Lists both integrations. |
| examples/vllm/CommunityToolkit.Aspire.Hosting.VLLM.ConsumerApp/Worker.cs | Demonstrates a chat request. |
| examples/vllm/CommunityToolkit.Aspire.Hosting.VLLM.ConsumerApp/Program.cs | Registers the example chat client. |
| examples/vllm/CommunityToolkit.Aspire.Hosting.VLLM.ConsumerApp/CommunityToolkit.Aspire.Hosting.VLLM.ConsumerApp.csproj | Wires consumer dependencies. |
| examples/vllm/CommunityToolkit.Aspire.Hosting.VLLM.AppHost/Properties/launchSettings.json | Defines local launch profiles. |
| examples/vllm/CommunityToolkit.Aspire.Hosting.VLLM.AppHost/CommunityToolkit.Aspire.Hosting.VLLM.AppHost.csproj | Connects hosting and consumer projects. |
| examples/vllm/CommunityToolkit.Aspire.Hosting.VLLM.AppHost/appsettings.json | Configures example logging. |
| examples/vllm/CommunityToolkit.Aspire.Hosting.VLLM.AppHost/AppHost.cs | Connects the GPU server and consumer. |
| examples/vllm/CommunityToolkit.Aspire.Hosting.VLLM.AppHost.TypeScript/tsconfig.json | Configures TypeScript compilation. |
| examples/vllm/CommunityToolkit.Aspire.Hosting.VLLM.AppHost.TypeScript/package.json | Defines TypeScript tooling and scripts. |
| examples/vllm/CommunityToolkit.Aspire.Hosting.VLLM.AppHost.TypeScript/package-lock.json | Locks JavaScript dependencies. |
| examples/vllm/CommunityToolkit.Aspire.Hosting.VLLM.AppHost.TypeScript/eslint.config.mjs | Checks asynchronous AppHost calls. |
| examples/vllm/CommunityToolkit.Aspire.Hosting.VLLM.AppHost.TypeScript/aspire.config.json | Configures the SDK and hosting reference. |
| examples/vllm/CommunityToolkit.Aspire.Hosting.VLLM.AppHost.TypeScript/apphost.mts | Demonstrates CPU hosting in TypeScript. |
| examples/vllm/CommunityToolkit.Aspire.Hosting.VLLM.AppHost.Go/go.mod | References generated Go modules. |
| examples/vllm/CommunityToolkit.Aspire.Hosting.VLLM.AppHost.Go/aspire.config.json | Configures the Go AppHost SDK. |
| examples/vllm/CommunityToolkit.Aspire.Hosting.VLLM.AppHost.Go/apphost.go | Demonstrates hosting APIs in Go. |
| CommunityToolkit.Aspire.slnx | Registers packages, examples, and tests. |
| CODEOWNERS | Assigns ownership of the integration. |
Review effort: Balanced
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Closes #1484
This adds an Aspire hosting integration for running vLLM and a client integration for consuming its OpenAI-compatible API. The APIs follow the existing Ollama/OllamaSharp builder pattern while supporting GPU configuration, Hugging Face model setup, and Microsoft.Extensions.AI
IChatClientregistration.In the AppHost:
In the consuming app:
The client appends
/v1to the hosting endpoint. The client model ID must match the name served by vLLM; Hugging Face credentials are only needed for gated models and should be supplied as a secret parameter.PR Checklist
Other information
Suggested review order
src/CommunityToolkit.Aspire.Hosting.VLLM— resource/connection properties, image defaults, shared-memory sizing, and NVIDIA/AMD/Podman runtime arguments.src/CommunityToolkit.Aspire.VLLM— settings precedence, default/keyed registration, model resolution, health checks, and telemetry opt-outs.tests-app-hosts/VLLM.AppHost, and C#/TypeScript/Go examples; finish with solution, CODEOWNERS, and README integration-table registration.API shape for review:
VLLMnaming, the integration-specificVLLMGpuVendor, and the fluentAddVLLMClient(name).AddChatClient()shape. Hosting and client are combined because they implement the same issue and share the consumer example. Issue #1484 is open and assigned; I found no explicit maintainer approval there, so this PR does not claim formal approval.Review follow-up: Removed the fabricated token from the public-model Go example, made repeated
WithModelcalls replace the model, and gave distinct keyed clients independent health checks. The corresponding review threads have been answered and resolved.Validation
Local checks used the repository-pinned .NET SDK 11.0.100 and matching Aspire CLI. NuGet audit was disabled because the local vulnerability-data endpoint was unavailable; npm dependencies were restored from a locally warmed offline cache.
dotnet build src\CommunityToolkit.Aspire.Hosting.VLLM\CommunityToolkit.Aspire.Hosting.VLLM.csproj --no-restore --configuration Release --verbosity quietnet10.0andnet11.0, no warnings.dotnet build src\CommunityToolkit.Aspire.VLLM\CommunityToolkit.Aspire.VLLM.csproj --no-restore --configuration Release --verbosity quietnet10.0andnet11.0, no warnings.dotnet build tests\CommunityToolkit.Aspire.Hosting.VLLM.Tests\CommunityToolkit.Aspire.Hosting.VLLM.Tests.csproj --no-restore --configuration Release --verbosity quietASPIRE012warnings from staged AppHost SDK references.dotnet test --project tests\CommunityToolkit.Aspire.Hosting.VLLM.Tests\CommunityToolkit.Aspire.Hosting.VLLM.Tests.csproj --no-restore --hangdump --hangdump-timeout 7m --verbosity quiet/v1/models; the TypeScript test waits for health.dotnet test --project tests\CommunityToolkit.Aspire.VLLM.Tests\CommunityToolkit.Aspire.VLLM.Tests.csproj --no-restore --verbosity quietgo test ./...in the Go AppHost examplegit diff --checkand outgoing credential/path scanCPU startup: The C# smoke host and TypeScript CPU example use
--enforce-eager, without changing integration defaults or limiting the sequence count. This disables graph compilation and JIT warmup for these low-resource workloads, potentially reducing inference throughput. With a warm model cache, two containers sharing four CPUs and limited to 8 GiB each reached health and listed models in 70.1s and 82.6s, then both served completions; PyTorch/oneDNN were constrained to AVX2. In the same local setup with one sequence, compiled execution took 267.2s and 299.3s with JIT warmup enabled, versus 147.4s and 168.0s with only JIT warmup disabled; both latter servers also served completions. This implicates JIT warmup as a significant startup cost, although the exact blocked worker frame in the earlier hosted timeout is not known.Hosted CI: The .NET CI workflow passed for commit
3c27e13c; the Ubuntu hosting suite passed all 19 tests with no skips in 2m 48s, including both CPU Docker hosts. Windows hosting and client tests, Ubuntu client tests, both builds, and CodeQL also passed. The remaining integration-table check fails withNETSDK1005, the repository-level issue tracked by open PR #2530.Limits and follow-ups: GPU/ROCm/Podman behavior is covered by app-model assertions, not a hardware/runtime test of this port. The C# CPU Docker smoke test checks readiness and model listing; the separate constrained experiment also exercised a completion. Go and TypeScript validate generated Aspire AppHosts over the same hosting integration; they are not alternate vLLM implementations. Full aspire.dev documentation belongs in a separate PR to microsoft/aspire.dev, as described in CONTRIBUTING.md. No full-repository test run is claimed.