Skip to content

Latest commit

 

History

History
369 lines (289 loc) · 17.9 KB

terminal-interface.mdx

File metadata and controls

369 lines (289 loc) · 17.9 KB

{/* SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. SPDX-License-Identifier: Apache-2.0 */}

Using the PAIR terminal interface

nvpair-tui is a full-screen keyboard-driven interface for operating one PAIR host and managing supported engine operations on paired cluster peers.

This is the headless option, not a second UI. Use it on a machine with no desktop environment, or over SSH, where the desktop application cannot run. If a desktop is available, use the desktop application.

It has known limitations. It is an operations tool, not a full replacement. It can list models, perform supported model and engine operations, and show workload destinations. Inference submission and some remote operations remain outside its scope. The full list is in What the Terminal Interface Cannot Do, and it is worth reading before you depend on it.

Never run the terminal interface and the desktop application on the same machine at the same time. This is not a preference. Each one starts its own broker and full worker tree, so two sets of services would fight over the same proxy ports, try to manage the same engines, and write the same settings files. Expect port conflicts, engines that refuse to start, and confusing state on both sides. Quit one before starting the other. This warning applies to dashboard mode, not nvpair engine, which starts no service tree.

Within its scope it is self-contained. It starts its own service process tree, so you do not run anything else first.

nvpair and nvpair-tui Commands

The nvpair wrapper runs nvpair-tui. With no subcommand, either opens the full-screen dashboard. The separate engine subcommand runs an engine's CLI without opening the dashboard:

nvpair engine                         # ENGINE, NAME, INSTALLED, SUPPORTED table and usage
nvpair engine ollama list
nvpair engine lmstudio --help
./nvpair-tui engine ollama --version  # No wrapper required

Keep nvpair-engine-manager beside nvpair-tui. The command asks that sibling for --list-engines or --resolve <engine>; it never searches PATH for an engine or guesses vendor locations. All arguments after the engine name pass through unchanged, even --version, -h, --, and quoted spaces. The CLI inherits stdin/stdout/stderr, working directory and environment, overlaid with the resolver's required environment. Ollama targets the effective engine port, not the PAIR proxy; Linux retains its library paths. Neither your shell's PATH nor its environment is changed.

Unlike dashboard mode, nvpair engine may run alongside the desktop app: resolution starts no broker or workers and writes no settings. The chosen CLI can still perform user-requested mutations. On Unix it replaces the launcher; on Windows it runs in the same console with inherited stdio. The engine's exit code is returned. Windows console/Ctrl-C behavior remains runtime-untested on this macOS verification host. Resolution or launch failures print one nvpair: <reason> stderr line and exit 2. An external HTTP service without a local executable cannot be launched; see engine CLI usage and fallback paths.

Do not count on nvpair existing on a headless machine. Only the Debian desktop package writes the wrapper at install time. On Windows and macOS the desktop application creates it at startup on a best-effort basis; it may never appear if you never open the packaged app. macOS skips it when /usr/local/bin is unwritable. Linux desktop startup tries ~/.local/bin, then /usr/local/bin, but does not ensure that the chosen directory is on your shell's PATH. Standalone service installers and source builds install no wrapper. Use the full path to nvpair-tui (with engine for CLI mode), or the engine full-path table. Refer to Building PAIR.

Start It

Before you start it, confirm the desktop application is not running on this machine, including minimized to the tray or menu bar. Quit it first.

nvpair-tui

From the release archive, extract it and run nvpair-tui from the directory the binaries are in. It looks for nvpair-ui-broker beside itself, so keep the extracted files together:

cd <directory containing nvpair-tui>
./nvpair-tui
Flag Effect
--broker-path <path> Use a service binary that is not beside nvpair-tui
--log-level <level> Verbosity of the terminal interface's own logging: debug, info, warn, or error. PAIR also reads this from NVPAIR_LOG_LEVEL
--version Print the version and exit

The interface's own log output goes to stderr, so it never corrupts the display. Service logs appear on the Logs tab instead.

Press q to quit. Quitting shuts the services down cleanly rather than leaving them running.

Keep It Running After You Disconnect

Quitting stops the services, and losing an SSH connection ends the program the same way. A node you reach over SSH therefore stops serving requests when your session drops. Start it inside a terminal multiplexer so the session survives independently of your connection:

tmux new -s pair
nvpair-tui

Detach with Ctrl-b d, and reattach later with tmux attach -t pair. The services keep running in between.

tmux and GNU Screen both work. With Screen, start with screen -S pair, detach with Ctrl-a d, and reattach with screen -r pair. Neither ships with PAIR. Install whichever your distribution provides.

The Screen

The top line shows NVPAIR TUI and the connection state on the right: connecting to broker..., broker ready v<version>, or broker disconnected. The second line is the numbered tab bar. The footer shows the keys available on the current tab.

If you see starting..., the interface is waiting for the terminal size. If the header stays on connecting to broker..., the service tree did not come up, and the Logs tab says why.

The terminal interface on the Overview tab, showing the header with broker ready, the numbered tab bar, the worker table, and the footer keys.

Moving Around

Key Action
tab, l, → Next tab
shift+tab, h, ← Previous tab
? Toggle full help
q, ctrl+c Quit

Within a table, j / k (or ↓ / ↑) move the selection, f / b page, and g / G jump to the first and last row.

While you are typing into a field, such as a PIN, an address, a port, or a model name, every key goes to that field. Press enter to submit or esc to cancel. Tab switching and q do not work until you do.

Tabs

# Tab What It Shows
1 Overview Broker uptime/version and all twelve workers: state, attempts, last error; R restarts with confirmation
2 Errors Active service errors by severity, age, node, and message
3 Nodes Nodes discovered on the network, with Connected or In cluster status
4 Proxies Ollama, LM Studio and oMLX proxies: listening port, discovered upstreams, and which node is selected
5 Workloads Live inference workloads: ID, model, engine, origin, destination, state, and age
6 Engines Selected node's engines: installed, running, healthy, port, and model inventory
7 Cluster This node's identity, cluster membership, and pairing
8 Manual Persistent addresses, PAIR/engine kind, verification and reachability
9 Settings Node settings: force ports, cluster auto-sync, and cluster ID and name
10 Logs Service log output, with live log-level control

Worker Health and Restart

Overview renders the broker's workers:get-status snapshot and full workers:update pushes, refreshed after an epoch change; it does not infer health from crash errors. States are running, degraded, restarting, exhausted, and missing. Select a worker and press R to request workers:restart after confirmation: y confirms; n, esc, or enter cancel. Other keys, including quit and tab switching, stay captured while the question is open. The request initiates recovery; later updates show the result.

The broker probes every 30 s with a 5 s deadline, degrades after two failures, and restarts eight eligible workers after three. Engine-manager and all three proxies never auto-restart on a hang. Restarting engine-manager may leave engines running or stop them; restarting a proxy interrupts inference through it. These warnings appear before confirmation. Missing defaults wait for explicit retry; exhausted workers need it to resume. See Health and Failure Detection for the shared retry budget and eligibility. Unix-only cross-process hang tests do not prove Windows recovery. Broker-client heartbeat/silence recovery is a separate layer, not this worker table's health policy.

Pair This Machine with Another

Pairing is the same six-digit PIN exchange the desktop application uses.

To invite a machine you can already see, from the Nodes tab (3):

  1. Select it with j / k.
  2. Press i. The status line shows the PIN.
  3. Read that PIN to whoever is at the other machine.

This is the easier path, because there is no address to type. Prefer it whenever PAIR has already discovered the machine you want.

To invite a machine by address, from the Cluster tab (7):

  1. Press i.
  2. Type the other machine's host, or host:port if it is not on the default pairing port.
  3. Press enter. The status line shows the PIN.
  4. Read that PIN to whoever is at the other machine.

Use this when discovery has not found the machine, for example on a network that filters multicast.

To accept an invitation from another machine, from the Cluster tab:

  1. Wait for the status line to read invite received from <name>.
  2. Press a.
  3. Type the PIN displayed on the inviting machine and press enter.

Press d instead to decline. After pairing, the peer appears under Members.

The terminal interface Nodes tab with a discovered machine selected and the pairing PIN on the status line.

Key Action Tab
i Invite the selected discovered node Nodes
i Invite by address Cluster
a Accept an inbound invitation Cluster
d Decline an inbound invitation Cluster
r Confirm removal of the selected member; successful unpair also forgets its manual PAIR addresses Cluster
L Confirm leaving the cluster (capital L) Cluster

On Manual, a adds a PAIR machine by hostname or IP and persists its address before starting PIN pairing; p retries pairing the selected PAIR entry. Unverified PAIR entries never route. e explicitly adds Engine only, a labelled manual bypass that trusts no PAIR claims; a PAIR owner's engine needs LAN engine binding enabled. r forgets a manual entry without removing cluster membership. Old entries without kind migrate to PAIR; remove/re-add a bare engine with e.

Only pair when you trust both machines and the network. The PIN is a short-lived bootstrap code, not a durable credential. Refer to the security policy.

Prepare an Engine and a Model

On the Engines tab (6), press n to cycle This node, paired discovered peers, and cluster members with a host UUID. The title identifies the target; offline members remain selectable, with connection or trust errors shown on the status line. Peer engine status is fetched on selection and after remote actions. Select an engine with j / k, then use:

Key Action Availability
n Next target node Local and peers
i Local: install (then s to start); peer: install and start Reported install capability
s / x Start / stop Local and peers
r Restart Local only
u Confirm uninstall Local; reported uninstall capability
U Confirm update (reinstall; capital U) Local; reported install and uninstall capabilities
e Edit engine server port, then confirm Local; reported set_port capability
m Open / close model inventory Selected engine
p Download a model Reported pull_model capability
d Confirm deletion of the selected model Models pane; reported delete_model capability

The engine-manager on the target node reports capabilities from its effective manifest, platform and ownership. Unsupported actions are omitted from help and their keys do nothing; absent capabilities from an older peer count as false. Capabilities describe support, not a guarantee that an operation will succeed. oMLX can list models, but cannot pull or delete them or edit its server port.

Press m to see model names and which are loaded. While the pane is open, j / k select models rather than engines; press m again to return. An absent inventory displays Model list unavailable (engine stopped or unsupported); a known empty inventory displays No models. Local inventory follows engine-manager snapshots and notifications. Peer inventory comes from discovery and may lag the peer; it is not a direct live query of that engine.

Pressing p opens a model-name prompt. Type, for example, qwen4:12b, and press enter. Local download progress appears on the status line. Remote pull and install complete on the RPC reply, not on a progress notification.

Port editing (e) accepts integers from 1 to 65535. After enter, confirm the warning that the engine will restart on that port if running. This changes the engine's persistent server port, not its proxy port.

Update is destructive reinstall, not an in-place upgrade. U warns that the engine is uninstalled, then installed and started: inference stops meanwhile, and files removed by its uninstaller are deleted. If uninstall fails, install is not attempted. The status line identifies the failing phase; there is no rollback or data-preservation guarantee.

Uninstall, update, port changes and model deletion require y to proceed; n, esc, or enter cancel. Other navigation and quit keys are captured while the confirmation is open. The question freezes the displayed target, so later list changes do not redirect it. Model deletion also names the node, engine and model; if delete_restarts_engine is true (bundled LM Studio), it warns that running inference will be interrupted. Cluster removal, cluster leave (L), and manual-node removal (r on Manual) use the same default-cancel prompt. If the local cluster identity changes while confirming leave, confirm again rather than leaving the replacement cluster.

A node can serve a request only when it is online, a compatible engine is running, and the requested model is present on that node. To route across several machines, download the same model on each.

Check Routing and Health

The Workloads tab (5) lists inference work as it runs, with its model, engine, ORIGIN, DESTINATION, state, and age. Origin is the submitting node; destination is the node assigned to execute it (scheduledOn), not proof that a still-pending request completed. Known nodes show their friendly name; unknown or departed nodes retain the first eight characters of their UUID, and an absent destination displays —. Failover updates the destination. The screenshot below predates the origin and destination columns.

The terminal interface Workloads tab listing live inference workloads with model, engine, state, and age.

The Proxies tab (4) shows each proxy's listening port and whether it is routing automatically (selected=auto) or pinned to one node. Press g to cycle Ollama, LM Studio and oMLX, enter to pin the highlighted upstream, and a to return to automatic routing. Leave it on automatic unless you are deliberately testing one node.

The Overview tab (1) reports the broker's worker registry and probe results; see Worker Health and Restart. These control-loop health checks do not prove engine HTTP readiness or successful inference.

Look at Errors and Logs

On Errors (2), press c to clear the selected entry.

On Logs (10), scroll with j / k and the page keys. Set the log level for the whole service fleet with d (debug), i (info), w (warn), or e (error). This is the first place to look when something has not started.

Change Settings

On Settings (9), move with j / k and press enter. Booleans toggle immediately. Text fields open for editing, with enter to save and esc to cancel.

Change proxy ports on the Proxies tab with p, not here. Refer to Ports for how PAIR arranges ports.

What the Terminal Interface Cannot Do

It is an operations tool, not a full replacement for the desktop application:

  • It has no built-in way to send an inference request. Use curl or another client against the proxy port, as in Getting Started.
  • It cannot uninstall, update or change the port of an engine on another node. Those operations are local-only; peer restart is not offered either.
  • It cannot pull or delete oMLX models: the engine offers neither action. Manage those files outside PAIR using oMLX's own workflow.

See Also

These documents cover related topics: