From c676edb79875780b36b901d09af890f22dd25faa Mon Sep 17 00:00:00 2001 From: meg miranda Date: Mon, 12 Oct 2026 13:47:45 -0700 Subject: [PATCH 1/2] AI aided technical edits for consistency across files and some review against the google release note file --- docs/engine-lifecycle.mdx | 8 ++++---- docs/inference-dispatcher.mdx | 13 +++++++++---- docs/overview.mdx | 16 ++++++++++------ docs/release-notes.mdx | 8 ++++++-- docs/upgrade-guide.mdx | 34 +++++++++++++++++++++------------- 5 files changed, 50 insertions(+), 29 deletions(-) diff --git a/docs/engine-lifecycle.mdx b/docs/engine-lifecycle.mdx index 545e499d2..1a2c705c4 100644 --- a/docs/engine-lifecycle.mdx +++ b/docs/engine-lifecycle.mdx @@ -166,16 +166,16 @@ After systems are paired, you can view another machine's engines from **Overview** and, where supported, start or stop engines and pull models on that peer. -Some actions stay **local only** on each machine: uninstalling, updating, and -changing an engine's CORS settings. Perform those on the machine that owns the +Some actions stay **local only** on each machine: uninstalling an engine and +changing its CORS settings. Perform those on the machine that owns the engine, or connect to that machine's PAIR application or terminal interface. Ports and startup arguments can be edited from a paired machine, as described in [Engine Settings](engine-settings.mdx). Everything you change is scoped to one machine. A port you set on this node moves that node's engine and nothing else. The other nodes keep the ports they already -had, and PAIR does not push the value across the cluster. Installing, updating, -and uninstalling work the same way, because each is a decision about one machine. +had, and PAIR does not push the value across the cluster. Installing and +uninstalling work the same way, because each is a decision about one machine. This is why the endpoint to use is the one shown on the machine you are working at, rather than a port you remember from somewhere else. diff --git a/docs/inference-dispatcher.mdx b/docs/inference-dispatcher.mdx index 0bf3471ed..cd377576d 100644 --- a/docs/inference-dispatcher.mdx +++ b/docs/inference-dispatcher.mdx @@ -3,7 +3,7 @@ SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All SPDX-License-Identifier: Apache-2.0 */} -# Inference dispatcher +# Inference Dispatcher `scripts/inference-dispatcher` is a standalone HTTP client for exercising Ollama, LM Studio, and llama.cpp-compatible servers. It deliberately has no @@ -38,7 +38,7 @@ both the desktop app and the terminal interface spawn once per request; see [architecture.mdx](architecture.mdx). The wrappers exist so the same tool can be driven by hand. -## Model selection +## Model Selection No model name is compiled into the program. When `--model` is omitted (or set to `auto`), the dispatcher queries the live model inventory, filters out models @@ -68,6 +68,11 @@ non-model defaults. } ``` +This sends four generated prompts, two at a time. To send prompts of your own, +list them in `prompts`, for example +`"prompts": ["Explain unified memory in two sentences."]`. One request is sent +per listed prompt, and `count` is ignored. + Pass the file with `--config path/to/config.json` or `INFERENCE_DISPATCHER_CONFIG`. Every scalar option also has an `INFERENCE_DISPATCHER_*` environment equivalent. @@ -77,7 +82,7 @@ Ollama, `1234` for LM Studio, and `8080` for llama.cpp. These are the ports the router's proxies claim, so requests enter the cluster router rather than a single engine. Use `--port` to select another local port. -## API behavior +## API Behavior - Ollama inventory: `GET /api/tags` - Ollama inference: `POST /api/generate` @@ -98,7 +103,7 @@ process cancellation. Parallel dispatch uses a configurable concurrency bound. `--result-log` appends JSON Lines records, while `--debug-error-log` writes a compact tab-separated error trail. -## What is never recorded +## What is Never Recorded Neither log, nor stdout, ever contains prompt text or response bodies. Prompts are identified by a `sha256: len=` digest and responses by their byte diff --git a/docs/overview.mdx b/docs/overview.mdx index 05f4e51c9..5f2cbdd1e 100644 --- a/docs/overview.mdx +++ b/docs/overview.mdx @@ -88,8 +88,9 @@ flowchart LR Node --> Engine["Inference engine"] ``` -Between machines, PAIR encrypts both the request out and the reply in, so the -whole round trip stays protected rather than the outbound half alone. Your +Between paired machines, PAIR encrypts both the request out and the reply in, so +the whole round trip stays protected rather than the outbound half alone. A node +you add manually without pairing it may be reached over plaintext HTTP. Your application sees an ordinary reply and never learns which machine served it, though you can see that yourself. @@ -121,7 +122,8 @@ Discovery is deliberately not a trust decision. Every machine on the network is visible. Pairing is the trust decision, and it is explicit and mutual. It requires an invitation and a PIN that a person enters. After pairing, node-to-node traffic runs over mutual TLS restricted to nodes in the cluster, -and the cluster refuses a machine that is not a member. +and the cluster refuses a machine that is not a member. The exception is a node +you add manually without pairing: PAIR may reach it over plaintext HTTP. The PIN is a short convenience code for bootstrapping that exchange, not a strong authenticator, so pair only over networks and with machines you trust. Local @@ -140,9 +142,11 @@ PAIR provides these capabilities: Messages API requests. - Pairing and cluster membership managed by the background services. - Model-aware, workload-informed routing of independent requests. -- Encrypted routing between machines: a request sent to another node travels over - mutual TLS restricted to the nodes you have paired, and the cluster refuses a - machine that is not a member. Local applications reach the proxy over loopback. +- Encrypted routing between paired machines: a request sent to a paired node + travels over mutual TLS restricted to the nodes you have paired, and the cluster + refuses a machine that is not a member. A manually added node that is not + paired may be reached over plaintext HTTP. Local applications reach the proxy + over loopback. - A desktop application, plus a terminal interface for headless machines, driving the same services. - Visibility into nodes, engines, models, workloads, and service errors. diff --git a/docs/release-notes.mdx b/docs/release-notes.mdx index c9a59bd5e..1c134cf16 100644 --- a/docs/release-notes.mdx +++ b/docs/release-notes.mdx @@ -28,7 +28,9 @@ its public development and contribution workflow on GitHub. settings such as context size where the engine supports them. Refer to [Engine Settings](engine-settings.mdx). - **RTX Spark support.** PAIR's Windows on ARM build has been validated with - llama.cpp and CUDA on RTX Spark N1X hardware. + llama.cpp and CUDA on RTX Spark N1X hardware. Windows on ARM + remains experimental. Refer to + [What is supported](../README.md#what-is-supported). - **Improved terminal experience.** The terminal interface has a redesigned layout organized around each device, so you can manage engines, download models, and pair devices from one place. A new **Jobs** tab follows inference @@ -48,7 +50,9 @@ its public development and contribution workflow on GitHub. - **LM Studio installed by an earlier PAIR.** It keeps working as before. PAIR now removes only engines it has a record of installing, and earlier versions kept no record, so uninstalling or resetting PAIR leaves that LM Studio in - place. You can remove it from `~/.lmstudio` yourself if you no longer want it. + place. To remove it while keeping your downloaded models, or to let PAIR + manage it from now on, follow the + [Upgrade Guide](upgrade-guide.mdx#if-an-earlier-pair-installed-lm-studio-for-you). The [Upgrade Guide](upgrade-guide.mdx) walks through each of these. diff --git a/docs/upgrade-guide.mdx b/docs/upgrade-guide.mdx index 9c2e5fd69..e256185fd 100644 --- a/docs/upgrade-guide.mdx +++ b/docs/upgrade-guide.mdx @@ -70,13 +70,15 @@ keep the models you downloaded. 1. If you also use the LM Studio desktop application, remove LM Studio through that application instead, since it shares the same `.lmstudio` folder. 2. Quit PAIR, which stops the LM Studio server it started. -3. Stop LM Studio and delete everything in `.lmstudio` except the `models` - folder. +3. Stop LM Studio, including its background process, and delete everything in + `.lmstudio` except the `models` folder. Stopping the server alone leaves that + background process running, and it can keep files open while you delete them. On macOS or Linux: ```bash ~/.lmstudio/bin/lms server stop + pkill -x lms; pkill -x llmster; sleep 2 find ~/.lmstudio -mindepth 1 -maxdepth 1 ! -name models -exec rm -rf {} + ``` @@ -84,6 +86,11 @@ keep the models you downloaded. ```powershell & "$env:USERPROFILE\.lmstudio\bin\lms.exe" server stop + $root = Join-Path $env:USERPROFILE '.lmstudio' + Get-CimInstance Win32_Process | + Where-Object { $_.ExecutablePath -and $_.ExecutablePath.StartsWith($root) } | + ForEach-Object { Stop-Process -Id $_.ProcessId -Force } + Start-Sleep -Seconds 3 Get-ChildItem "$env:USERPROFILE\.lmstudio" -Force | Where-Object Name -ne 'models' | Remove-Item -Recurse -Force @@ -104,20 +111,21 @@ and desktop applications are not affected. PAIR's local endpoints now use each engine's own browser access setting, known as CORS, rather than allowing every website. That means a web page can reach your models through PAIR only if you have allowed it in the engine. If a -browser-based client stopped connecting after the update, add its address in -the engine's launch options: +browser-based client stopped connecting after the update, allow it in the +engine's launch options. Ollama and llama.cpp take a list of allowed addresses; +LM Studio has an on/off switch instead: | Engine | Setting | | --- | --- | -| Ollama | `OLLAMA_ORIGINS`, for example `OLLAMA_ORIGINS="https://example.com"` | -| LM Studio | The `--cors` switch | -| llama.cpp | `--cors-origins` | - -Make the change on the machine that runs the engine. Include the address's -scheme, such as `https://`; a wildcard on its own (`*`) or a bare `localhost` -is not accepted. If requests can be routed to other machines in your cluster, -allow the address on each of them. [Engine Settings](engine-settings.mdx) has -the details. +| Ollama | Add the address to `OLLAMA_ORIGINS`, for example `OLLAMA_ORIGINS="https://example.com"` | +| LM Studio | Add the `--cors` switch. It takes no address; LM Studio decides which origins it allows. | +| llama.cpp | Add the address to `--cors-origins` | + +Make the change on the machine that runs the engine. For Ollama and llama.cpp, +include the address's scheme, such as `https://`; a wildcard on its own (`*`) +or a bare `localhost` is not accepted. If requests can be routed to other +machines in your cluster, make the change on each of them. +[Engine Settings](engine-settings.mdx) has the details. ## Checking Everything Is Working From 219adb349bd20ccdd1ba58a91de50aef88df3c15 Mon Sep 17 00:00:00 2001 From: meg miranda Date: Tue, 13 Oct 2026 10:51:23 -0700 Subject: [PATCH 2/2] removed notes about manually added nodes using http Signed-off-by: meg miranda --- docs/overview.mdx | 11 ++++------- 1 file changed, 4 insertions(+), 7 deletions(-) diff --git a/docs/overview.mdx b/docs/overview.mdx index 5f2cbdd1e..2053efcb8 100644 --- a/docs/overview.mdx +++ b/docs/overview.mdx @@ -89,8 +89,7 @@ flowchart LR ``` Between paired machines, PAIR encrypts both the request out and the reply in, so -the whole round trip stays protected rather than the outbound half alone. A node -you add manually without pairing it may be reached over plaintext HTTP. Your +the whole round trip stays protected rather than the outbound half alone. Your application sees an ordinary reply and never learns which machine served it, though you can see that yourself. @@ -122,8 +121,7 @@ Discovery is deliberately not a trust decision. Every machine on the network is visible. Pairing is the trust decision, and it is explicit and mutual. It requires an invitation and a PIN that a person enters. After pairing, node-to-node traffic runs over mutual TLS restricted to nodes in the cluster, -and the cluster refuses a machine that is not a member. The exception is a node -you add manually without pairing: PAIR may reach it over plaintext HTTP. +and the cluster refuses a machine that is not a member. The PIN is a short convenience code for bootstrapping that exchange, not a strong authenticator, so pair only over networks and with machines you trust. Local @@ -144,9 +142,8 @@ PAIR provides these capabilities: - Model-aware, workload-informed routing of independent requests. - Encrypted routing between paired machines: a request sent to a paired node travels over mutual TLS restricted to the nodes you have paired, and the cluster - refuses a machine that is not a member. A manually added node that is not - paired may be reached over plaintext HTTP. Local applications reach the proxy - over loopback. + refuses a machine that is not a member. Local applications reach the proxy over + loopback. - A desktop application, plus a terminal interface for headless machines, driving the same services. - Visibility into nodes, engines, models, workloads, and service errors.