Repository navigation
Conversation
Lifecycle fixes needed by an engine that takes a minute or more to start and forks helper processes, such as vLLM. None of them is engine-specific. - engine:status and engine:get-installed no longer wait for a running lifecycle operation, so a toggle during a slow start does not time out. - Shutdown caps the stop grace at 3s so the force kill lands inside the parent's window, and a SIGTERM no longer blocks on an idle stdin. - A stop also kills helper processes left in the engine's process group, so a crash mid-load does not orphan a process holding GPU memory. - An orphaned engine run by an interpreter (a Python package's script) is recognised by its command line, so the next launch can stop it. Signed-off-by: Tushar Khinvasara <tkhinvasara@nvidia.com>
…ache Add an hf_cache manifest action that lists the models in a Hugging Face hub cache directory. It reads the disk, so it works while the engine is stopped, which lets an engine that serves one model per process (vLLM) show every model it could be switched to. A repository is listed only when its snapshot is a complete, text-model-shaped download: vision-only and GGUF-only repositories, interrupted downloads and links whose blob is gone are left out. Signed-off-by: Tushar Khinvasara <tkhinvasara@nvidia.com>
Add a launch_model manifest action for an engine that has no load call because
the model is a launch argument (vLLM). It takes {"model": "<id>"}, writes the
model into the saved launch settings under the declared flag, and restarts a
running engine on it. If the new model does not come up, the previous one is
restored and the error says so. A stopped engine only has its saved launch
changed, and asking for the model already in the launch arguments restarts
nothing.
The action requires editable launch settings and a readiness probe on every
platform, and refuses model ids that could be read as a flag, a path or a
shell fragment.
Signed-off-by: Tushar Khinvasara <tkhinvasara@nvidia.com>
Register vllm alongside Ollama, LM Studio and llama.cpp: facade port 8000, engine port 8001, discovery key "vl". The proxy and broker treat it like any other managed engine, and the terminal interface gets its model wire. The engine-manager manifest (Linux amd64/arm64) is detect-first: an existing vllm is adopted, otherwise a sha256-pinned uv installs vllm 0.32.0 into a private venv after checking for an NVIDIA driver with CUDA 13+ and enough disk. vLLM serves one model per process, so its list_models is an hf_cache action, load_model is a launch_model action, and a new manifest field, advertised_models: loaded, advertises only the served model to peers and the proxy, since a request for any other cached model would fail at the engine. Signed-off-by: Tushar Khinvasara <tkhinvasara@nvidia.com>
Check the bundled vLLM manifest's install, launch and model actions, that the install refuses drivers without CUDA 13, and, across real binaries, that the broker starts a vLLM proxy and the facade serves the model list and routes exact model ids. Signed-off-by: Tushar Khinvasara <tkhinvasara@nvidia.com>
Show vLLM next to the other engines: type, display name, links, welcome-screen default (Linux only), proxy identity and port state, icon and downloaded-model matcher. The icon is an original lettermark, not the vLLM project logo. vLLM serves one model per process and has no pull, eject or delete call, so a new hasPullModel capability hides Add model for it. Its model list is the Hugging Face cache and Load switches the served model by restarting vLLM; that load is observed so a failure is attributed to the model row, and the row stays locked for up to 20 minutes while the engine restarts. Document the engine across the desktop and top-level docs, tick the roadmap item, and add the vLLM model store to the data-reset notes. Signed-off-by: Tushar Khinvasara <tkhinvasara@nvidia.com>
Models with a 256K default context need more KV cache than the default 0.5 GPU memory share provides and fail to start. Pass --max-model-len 32768 by default and document it. Signed-off-by: Tushar Khinvasara <tkhinvasara@nvidia.com>
…ts it vLLM serves one model per process, so loading another restarts the engine. Show 'starting' instead of 'stopped', keep the model list, and await the load call so the switch does not look like the engine was disabled. Signed-off-by: Tushar Khinvasara <tkhinvasara@nvidia.com>
…lling per model vLLM refuses a context above the model's own limit, reserves 92% of the GPU by default, and rejects tool_choice auto without a tool-call parser. Agent clients such as Hermes need a 64K window and tool calling. Add a manifest auto_launch (hf_context) step that reads the served model's config.json and the free GPU memory at every start and appends --max-model-len (64K target, shrunk to fit), YaRN --hf-overrides when the model's window is shorter, --gpu-memory-utilization sized to the weights and KV cache, and --enable-auto-tool-choice with the hermes parser for Qwen families. Arguments the user saved always win, nothing is written back to saved settings, and tensor/pipeline-parallel launches are left alone. Drop the fixed 0.5 GPU share and 32K cap from the default launch. Signed-off-by: Tushar Khinvasara <tkhinvasara@nvidia.com>
…for vLLM vLLM has no pull API, so a model not already cached could only be used by naming it in the launch arguments. Add an hf_pull action kind for pull_model: it reads the repo's file list and sizes from Hugging Face, selects the files vLLM reads, and runs the hf tool from the engine's virtualenv, reporting progress from the bytes landing in the hub cache. It needs no running engine and stops when cancelled or stalled. Add a vLLM engine:catalog source: open, text-generating safetensors repos of a few publishers (text-generation and image-text-to-text), with parameter count and size, plus a searchable query like llama.cpp's. Signed-off-by: Tushar Khinvasara <tkhinvasara@nvidia.com>
Enable hasPullModel for vLLM, give it an engine hub, let the model hub search the upstream catalog for any engine that reports it (a new hubSearch capability, set for llama.cpp and vLLM), and warm the vLLM catalog with the others. Update the docs. Signed-off-by: Tushar Khinvasara <tkhinvasara@nvidia.com>
vLLM had no delete, so a cached model could only be removed by hand. Add an hf_cache delete variant for delete_model: it removes a model's folder and download locks from the hub cache, needs no running engine, refuses a path outside the cache, and refuses the model the engine is set to serve (running or not), failing closed when that cannot be determined. Enable hasDeleteModel for vLLM in the desktop, with a new deleteBlockedWhileLoaded capability that disables Delete on the loaded model. Update the docs. Signed-off-by: Tushar Khinvasara <tkhinvasara@nvidia.com>
vLLM cannot drop its model while running, so Eject (unload_model) is a new stop_engine action that stops the engine and frees all of its GPU memory. Load (launch_model) on a stopped engine now saves the choice and starts it, restoring the previous choice if it does not start. Enable hasEject for vLLM and keep its model list, Add model and Load available while stopped, since the list is read from disk. Delete removed only a model's folder, but recent Hugging Face tooling keeps large files in a shared hub/blobs store linked from each repo, so the weights stayed on disk. Delete now also removes shared files no other model uses, with their sidecars, and reports the bytes freed. Signed-off-by: Tushar Khinvasara <tkhinvasara@nvidia.com>
Add an optional runtime front: PAIR listens on the engine's public port and runs the engine itself on a loopback port at a fixed offset, forwarding requests unchanged, streaming responses, cancelling the engine's request when the client leaves, and answering 503 while the engine is not up. Only the launch that is spawned is repointed; the launch users see and validate keeps the public port. A stale engine on the internal port from a crashed run is reclaimed at the next start. This is the base for idle unload and wake on request. Only vllm.json declares a front, and the hooks in shared lifecycle code do nothing without one; a test checks that every other engine's launch is unchanged. Signed-off-by: Tushar Khinvasara <tkhinvasara@nvidia.com>
An engine behind a gateway now tracks inference activity at the gateway (not GET/HEAD/OPTIONS, so health checks and the model list do not count, and a request in flight keeps the engine loaded). After front.idle_timeout_s (600 s for vLLM, 0 never) with none, the manager stops the engine's process, freeing the GPU, while the gateway and the ON state stay. The status reports idle, /health still answers, inference is refused with 503, and the model it is set to serve is still advertised with nothing marked loaded. Eject on such an engine now unloads and keeps it ON like the other engines, and Load of the model it is set to serve loads it again; a failed load leaves the engine idle. Waking on an incoming request follows. Signed-off-by: Tushar Khinvasara <tkhinvasara@nvidia.com>
A request that needs the model and arrives while the engine is unloaded now loads it behind the gateway and is answered once it is ready, as Ollama and llama.cpp do, instead of getting a 503. Requests that arrive together share one start, a client that leaves does not abandon the load, and Eject followed by a request reloads it. A failed load answers 503 with the reason and leaves the engine idle so the next request retries. Health checks, the model list and other reads never load the model. Signed-off-by: Tushar Khinvasara <tkhinvasara@nvidia.com>
…e proxy The proxy routes by each node's advertised models and builds its own model list from the engine's /v1/models. While the engine was unloading or loading, the inventory briefly lost the model, and the idle gateway refused the model list, so a request arriving then was answered 'no node advertises the requested model' and never reached the wake. Record the served model when the engine goes idle, mark the state idle before the gateway so no reader sees an engine that is neither loaded nor idle, keep the engine marked running and idle while it loads (no stopped flicker), and answer GET /v1/models from the gateway while unloaded or loading. Signed-off-by: Tushar Khinvasara <tkhinvasara@nvidia.com>
…or loading The health check kept probing /health while the engine loaded again after an idle unload. The gateway forwards that probe to the engine for the start's readiness check, so it failed for the whole load and raised 'failed its health probe', which cleared itself once the load finished. Skip the probe while the engine is idle or loading, and test that no warning is raised while one stands. Signed-off-by: Tushar Khinvasara <tkhinvasara@nvidia.com>
…out settable Start, restart and the restore after PAIR restarts now bring an engine with a front up idle, with nothing loaded, as Ollama and llama.cpp start a server: about a second, the engine ON and healthy, the model loaded by the first request or a Load. The launch is still validated and the ports checked, so a launch that cannot work is refused at Start. Applying launch settings and switching or loading the model still load it at once, so a bad setting is found and rolled back. Add get_idle_timeout and set_idle_timeout (front_idle) actions: the limit is read and changed in seconds, 0 for never, applies at once and is saved in the engine's override file. Signed-off-by: Tushar Khinvasara <tkhinvasara@nvidia.com>
The status of an engine that is on with nothing loaded now carries idle, and the card reads 'Idle · loads on next request'. For vLLM the model row's expiry control becomes the idle limit (1 minute to 1 hour, or stays loaded), read from and sent to the engine manager as seconds through get_idle_timeout and set_idle_timeout; a limit that is not offered, such as one set by editing the override file, shows as the nearest one. A new engine-level list marks which engines use the expiry this way, and a test keeps it and the capability in step. Update the docs. Signed-off-by: Tushar Khinvasara <tkhinvasara@nvidia.com>
Unload an idle vLLM after five minutes, not ten, and show five minutes as the limit until the engine manager reports the real one. Update the docs. Signed-off-by: Tushar Khinvasara <tkhinvasara@nvidia.com>
Add a darwin/arm64 platform to the vLLM manifest. The install fetches the sha256-pinned uv, creates a managed Python 3.12 venv and installs the pinned vLLM 0.32.0 core and vLLM-Metal wheels, each checked against a digest. It refuses early on non-arm64, macOS older than 15, or under 10 GB free. Allow the install on macOS in the desktop engine capabilities, update the tests that assumed Linux only, and document the new platform. Signed-off-by: Tushar Khinvasara <tkhinvasara@nvidia.com>
vLLM serves one model per process, so a request naming another downloaded model used to be refused: only the loaded model was advertised. The gateway in front of vLLM now reads the request's JSON "model" field and, for a different downloaded model, lets the requests already running finish, unloads the model, saves --model, loads the new one behind the same gateway, and answers the held request. Requests that arrive together share one switch. A model that is not downloaded gets a 404 and nothing changes. A switch that fails is not rolled back: the request gets a 503 naming the model and the reason, and vLLM is left unloaded and ON, so the next request loads again. vLLM now advertises every cached model, including while idle. The gateway's /v1/models names them all, and loaded_models asks for /v1/models?resident=1 so only the loaded one is reported as resident. Signed-off-by: Tushar Khinvasara <tkhinvasara@nvidia.com>
- Re-check for a switch under the gateway lock before admitting a request, so a request cannot start on a model that is being unloaded or load the old one while the switch runs. - Verify the current model was really unloaded before saving the new one, and wait out a load that was only just starting. - Leave a name that is not a downloaded model (such as a --served-model-name alias) to the engine instead of answering 404. - Read the downloaded-model list once per interval for all requests, keep the last good list when a read fails, and bound the model-list body separately from the request peek. Signed-off-by: Tushar Khinvasara <tkhinvasara@nvidia.com>
95d8ec6 to
7c0f178
Compare
Several docs described vLLM as it was before later commits in this series: - vLLM also runs on Apple Silicon macOS, not only Linux. - Every cached model is advertised, and a request for one that is not loaded switches vLLM to it; the docs still said only the served model was advertised and other requests were refused. - The PAIR gateway listens on 8001 with vllm serve behind it on loopback 18001; the docs still put vllm serve on 8001 and said vLLM had no load, unload, pull or delete actions. Also document GET /v1/models?resident=1, and say "vLLM engine" rather than "vLLM backend" in the services parity table. Signed-off-by: Tushar Khinvasara <tkhinvasara@nvidia.com>
Add a probe that asks the installed vLLM which of its tool-call parsers reads a model's tool-call format. It runs with vLLM's own Python, renders the model's chat template with one example tool call, and offers that text to every registered parser; a parser matches when it extracts exactly that call. A tokenizer that ships response_template metadata gets vLLM's hf parser. Nothing is downloaded and no code from the model repository runs. Among parsers that read a format equally, the general-purpose ones are preferred. Results are kept per vLLM binary and snapshot for the life of the service; a failed probe is tried again on the next start. Signed-off-by: Tushar Khinvasara <tkhinvasara@nvidia.com>
Tool calling was enabled only for the Qwen families, with the hermes parser, so every other model answered HTTP 400 to agent clients, which send tools with tool_choice "auto" on every request. Replace the family table with the probe: a vLLM launch now gets --enable-auto-tool-choice and the parser that reads the model's own tool-call format, or no tool flags when none does. A parser named in the launch arguments is still used as given. Checked against vLLM 0.32.0: Qwen2.5 and Qwen3 get hermes as before; Qwen3.5 and Granite 4.2 now get qwen3_coder; Nemotron Nano v2 and Qwen2.5-VL get none. Signed-off-by: Tushar Khinvasara <tkhinvasara@nvidia.com>
… parser A model that no vLLM parser reads (Nemotron Nano v2, or a template with no tool support) still failed every chat from an agent client: vLLM rejects tool_choice "auto" without a parser, and such clients offer their tools on every request. The gateway now knows whether the running launch parses tool calls. When it does not, it removes tools, tool_choice and parallel_tool_calls from chat requests that leave tool_choice unset or "auto", so the model answers in plain text, and marks the response with X-PAIR-Tool-Calls. A request with tool_choice "required" or a named function is forwarded unchanged and gets vLLM's own error. Bodies are still never logged. Update the engine-manager README and the engine settings guide for the parser choice and this fallback. Signed-off-by: Tushar Khinvasara <tkhinvasara@nvidia.com>
kjlubick
left a comment
kjlubick
left a comment
There was a problem hiding this comment.
waitsForEngineReadiness in nvpair-engine-manager/remoteclient.go has a branch for ollama that might also be needed for vllm with regards to remote vLLM loads being canceled too early
| settings "nvpair-shared/enginesettings" | ||
| ) | ||
|
|
||
| // Front puts a PAIR-owned HTTP gateway on an engine's public port and runs the |
There was a problem hiding this comment.
Could you explain the architectural decision to add the inference gateway in nvpair-engine-manager/front.go? From my understanding, features like inference proxying should be handled by nvpair-proxy, or at least that's what the engine-manager spec says.
There was a problem hiding this comment.
I want to +1 this question. I don't understand the need for another proxy.
There was a problem hiding this comment.
Good question, and you're both right: this conflicts with the spec. nvpair-engine-manager/spec.md puts inference traffic out of scope ("this service never proxies /api/chat etc."), and the gateway does exactly that. I updated the README but not the spec, and I should have raised the design before building on it.
Why it's there: vLLM serves one model per process, loads it at startup, has no load/unload API, and holds most of the GPU until the process exits. The other engines can load a model on the first request, free memory when idle, and serve any model by name. To give vLLM the same behaviour, something has to see each request and start, stop or relaunch the process. I put that next to the lifecycle code in the engine manager, but in hindsight it's the wrong layer. It also has a practical gap: a request that triggers a large model load can outlast the proxy's 120 s response timeout.
Proposal: take the gateway out of this PR.
- This PR lands vLLM as an engine the user manages directly. Load restarts vLLM with the new model, and Eject stops it and frees the GPU. Hugging Face pull/list/delete, desktop support and tool-call parser selection stay; tool-call parsers are chosen at launch, with no request rewriting.
- Removed from this PR:
front.go, idle unload, wake on request, switch on request, removingtoolsfrom requests, and the idle-limit setting. Only the loaded model is advertised for routing. - On-demand loading gets a separate design discussion. The likely shape: the engine manager detects idleness from vLLM's own
/metrics, with no traffic through it, and the vLLM facade innvpair-proxy, which already sees every request, holds a request and asks the engine manager through the broker to load the model. vLLM's sleep mode is worth evaluating there too.
Does that direction work for you, or would you prefer something else? I'll hold off on the rework until we agree.
There was a problem hiding this comment.
@Noah-Tervalon-Nvidia, the reply above covers your +1 too. In short: the gateway was added so vLLM could load and unload models on demand like the other engines, but it doesn't belong in the engine manager, so I've proposed removing it from this PR and leaving Load and Eject to restart and stop vLLM directly. Are you OK with that direction? I'll hold off on the rework until you and @kjlubick agree.
There was a problem hiding this comment.
I like the direction of scoping this down. I also don't know that we should even be doing the gateway at all (even in another PR). It seems like by doing that we start to lose the core idea that we are just a passthrough. That gateway doing work of managing what's utilized etc. is basically a wrapper around vllm managing it, which seems outside of our scope.
engine:get-installed listed every manifest, and reported an engine with no block for the host's OS and CPU, such as vLLM on Windows, as an empty not-installed entry. The desktop and TUI then offered an install that could only fail. Leave such an engine out of the listing, and make engine:status return an error for it. Signed-off-by: Tushar Khinvasara <tkhinvasara@nvidia.com>
A node's engine manager now leaves out an engine the node cannot run. Once a node has listed its engines, mark the missing ones unsupported and give them no row in the engine list, tray, endpoints or first-run setup. This hides vLLM on Windows, while a Linux node viewed from Windows still shows it. Signed-off-by: Tushar Khinvasara <tkhinvasara@nvidia.com>
11269f5 to
a0eff8f
Compare
Noah-Tervalon-Nvidia
left a comment
Noah-Tervalon-Nvidia
left a comment
There was a problem hiding this comment.
I reviewed everything up through 4e8d2f4 (where you add the front.go). Since I don't really understand the rationale for adding that I skimmed the rest and they all rely on that so going to hold off any further comments.
|
|
||
| // hfShardRe matches the shard names Hugging Face repos use when they carry no | ||
| // index file, such as model-00003-of-00004.safetensors. | ||
| var hfShardRe = regexp.MustCompile(`^.+-(\d+)-of-(\d+)\.(?:safetensors|bin)$`) |
There was a problem hiding this comment.
NIT - my regex optimization is going off here, matching the start of a line and then .+ (or .*) is going to make it match every line that it's run against and then check every character before it eventually finds it doesn't match the rest of the string and give up. You can just start with the -(\d+)
There was a problem hiding this comment.
Agreed. The leading ^.+ is unnecessary, so the pattern is now -(\d+)-of-(\d+)\.(?:safetensors|bin)$. Since it only runs on individual file names and Go's RE2 engine does not backtrack, there was no measurable cost, but the shorter pattern states the intent more clearly.
| // arguments already say the same thing, whatever their quoting: the launch text | ||
| // shown to a user is quoted canonically, so comparing text would call an | ||
| // unchanged launch changed. | ||
| func withLaunchFlag(text, flag, value string) (result string, changed bool, err error) { |
There was a problem hiding this comment.
is there a reason that we are not using json to control the launch args similar to how we do it with the other engines? Then this parsing gets a lot easier by just searching the json file for the specific arg and updating it rather than this looping check doing string parsing.
There was a problem hiding this comment.
The launch arguments are stored as JSON, the same as for the other engines: the saved settings hold them as an array ("launch_args": ["--model", "Qwen/Qwen3.5-4B"]). The text form is the settings editor's representation, shared by every engine with editable launch settings. launch_model saves through that same path, so the host, port and CORS checks apply and the saved settings always match what the engine runs.
The loop works on the parsed argument list rather than on raw text. It is needed because command-line arguments are ordered rather than key/value: it has to handle both --model X and --model=X, a flag given more than once, and a -- terminator, while leaving every other argument in place. A JSON object could not represent valueless or repeated flags faithfully.
You are right that it could be simpler, though. I've folded the set and remove cases into one helper that drops every form of the flag, so setting the model is now "remove, then insert at the same position", with no behaviour change.
| settings "nvpair-shared/enginesettings" | ||
| ) | ||
|
|
||
| // Front puts a PAIR-owned HTTP gateway on an engine's public port and runs the |
There was a problem hiding this comment.
I want to +1 this question. I don't understand the need for another proxy.
…tore the previous model A remote load used the ordinary 30 s response-header budget for every engine except Ollama. Loading a vLLM model restarts vLLM on it, which can take minutes, so the caller gave up, the peer's handler was cancelled mid start, and the rollback to the previous model ran under the same cancelled context and failed at once, leaving vLLM down with the new model saved. Remote vLLM loads now use the readiness-sized client, and the previous model is restored under a context the caller cannot cancel. A test cancels a slow switch and checks that the previous model is back and running. Signed-off-by: Tushar Khinvasara <tkhinvasara@nvidia.com>
Replying to this review point. @kjlubick Thanks, good catch. Confirmed and fixed in 5c27d65. For vLLM, The fix:
Verified across two clustered machines with a real vLLM (RTX 6000 Ada → RTX A6000 over LAN mTLS):
|
167fedb to
13666dc
Compare
(review) Thank you for the detailed review. I've replied to each of your comments in its own thread, and the resulting changes are in the last commit. On |
|
Follow-ups from the review, kept out of this PR To keep this PR focused, these improvements came out of the review but are deferred. I'll open an issue for each once the PR merges.
Pending in this PR: removing the vLLM gateway ( To verify before merge: the Qwen3.5-4B default on Apple Silicon through vLLM-Metal, and launch sizing on a DGX Spark. Let me know you thoughts / comments on this |
- Match Hugging Face shard file names without a leading ^.+ in hfShardRe. - Drop the redundant shard-count comparison; the loop over shard numbers is the real completeness check, and a test now covers shards numbered from zero. - Allow deleting the model a stopped vLLM is set to serve. The delete takes --model out of the saved launch arguments, and an engine with a launch_model action refuses to start while its launch names no model, rather than letting vLLM download a default of its own. Loading a model selects one again. The model a vLLM process serves is still refused; an engine ejected behind its gateway is stopped first, so Eject then Delete works. - Say in hfpull.go that the download is done by Hugging Face's hf CLI, and state in the pull_model description and README that it fetches the files vLLM loads rather than the whole repository. - Fold setting and removing a launch flag into one helper, dropLaunchFlag. - Make Qwen/Qwen3.5-4B the default vLLM model in place of Qwen2.5-0.5B, state that it needs a GPU with at least 16 GB, and raise the install's free-disk check to cover the engine and that model (25 GB on Linux, 20 GB on macOS). - Size vLLM launches on unified-memory machines such as DGX Spark, where nvidia-smi reports [N/A] for GPU memory, from /proc/meminfo, as nvpair-node-info already does. Signed-off-by: Tushar Khinvasara <tkhinvasara@nvidia.com>
13666dc to
74f8e14
Compare
Resolve two conflicts with develop: - services/nvpair-engine-manager/README.md: keep develop's shutdown-sweep paragraphs and this branch's signal and engine:status paragraphs. - services/nvpair-ui-broker/enginesettings_test.go: use develop's reserveSettingsPort for the harness ports, keeping the fifth port this branch added for the vLLM proxy. Signed-off-by: Tushar Khinvasara <tkhinvasara@nvidia.com>
Description
Adds vLLM as a supported engine alongside Ollama, LM Studio, and llama.cpp, and lets it serve any model already downloaded in the Hugging Face cache on request.
/v1/modelson the engine lists all cached models, while the desktop marks only the loaded model as running.toolswithtool_choice: "auto"on every request, which vLLM rejects with HTTP 400 unless it was started with a tool-call parser. PAIR now picks the parser from the model itself: it renders the model's chat template with one example tool call and keeps the installed vLLM's parser that reads it back exactly (hfwhen the tokenizer shipsresponse_template). No list of model families is kept, nothing is downloaded, and no code from the model repository runs. When no parser fits, the gateway removestoolsfrom chat requests that leavetool_choiceunset or"auto", so the model answers in plain text instead of failing, and marks the response withX-PAIR-Tool-Calls. A request withtool_choice: "required"or a named function is forwarded unchanged.Other engines keep their current model and request behavior. Shared code changes in these ways:
engine:statusandengine:get-installedno longer wait for a running lifecycle operation.hf_cacheandlaunch_model, and the gateway (front), which only the vLLM manifest uses.engine:get-installedleaves out an engine whose manifest has no block for the host's OS and CPU, where it used to report an empty not-installed entry, andengine:statusreturns an error for it. The desktop marks an engine missing from a node's listing asunsupportedand gives it no row. Today this affects only vLLM.commitLaunchLockedis extracted fromConfigureLaunchwith unchanged logic.Release intent
Changelog title
vLLM engine support with on-request model switching
Changelog body
Bumps
Scope
Included: the vLLM engine (manifest, installer, gateway, auto-sizing, idle handling), request-driven model switching, desktop UI, proxy/broker/TUI wiring for the new engine, docs.
Excluded: loaded-node preference in
nvpair-proxy(cross-engine, separate change); fixing vLLM memory sizing for some multimodal/quantized models.This is a large PR (about 9,300 added lines across 116 files), so it is organised as 32 focused commits that can be reviewed one at a time. The groups below are in commit order:
hf_cachemanifest action, which lists Hugging Face cache models.launch_modelmanifest action, which switches an engine's model through its launch flag.Validation
go test ./...inservices/nvpair-engine-manager,services/nvpair-proxyand the other touched modules; race tests repeated 15 times on the gateway switch tests.npm run lint,typecheck,test:unit,dead-code:check,service-contracts:checkindesktop/.go test ./...inservices/tests(cross-process, real binaries), run with no other PAIR instance on the machine.node scripts/spdx-headers.mjs./v1/modelslists all cached models, UI marks only the loaded one, a request for another cached model swaps in.get_weathercall withqwen3_coder(also when streaming), Qwen3-8B withhermes; Nemotron Nano 9B v2 answers 200 in plain text with the tools removed, and withtool_choice: "required"gets vLLM's 400.--modelcleared, a start is refused with no download, and loading another model starts vLLM and answers a chat request.NVPAIR_LIVE_VLLM_BIN=… go test -run TestToolParserProbe_FindsTheParserWithTheInstalledVLLM).Checklist
git commit -s), certifying the Developer Certificate of Origin.services/versions.jsonis written by automation — do not edit it by hand.