Retry silent requests and temporarily deprioritize stalled nodes

README.md

@@ -240,6 +240,13 @@ The reply is ordinary OpenAI-shaped JSON, abbreviated here:
 If you changed a port, or you are using LM Studio or oMLX rather than Ollama, copy the
 URL from **Endpoints → API endpoints** instead of assuming the one above.
 
+The proxies keep the `120s` response-header timeout, but impose **no first-body
+deadline by default** after headers arrive. Long-context prefill can legitimately
+be silent for several minutes. `--first-body-timeout` /
+`NVPAIR_PROXY_FIRST_BODY_TIMEOUT` is a separate opt-in; see
+[Troubleshooting](docs/troubleshooting.mdx#requests-are-silent-after-headers)
+before enabling it.
+
 That is a single machine working. To route across machines, pair a second one
 from **Settings → Cluster** and repeat the engine and model steps there. The
 inviting machine shows a six-digit PIN, and you enter that PIN on the machine you

docs/architecture.mdx

@@ -476,6 +476,15 @@ see the same idle node and pile onto it. To prevent that, each proxy adds the
 requests it has just dispatched itself to its own estimate and reserves its
 choice before forwarding, so a burst spreads without waiting for those reports.
 
+After those preferences, each proxy moves candidates that just spent a full
+silent response-header deadline (or an explicitly enabled first-body deadline)
+behind their eligible peers, preserving relative order within both groups. This measured, process-local cooldown overrides pins,
+reservations, and prefix hints; it is not a liveness verdict. It never removes a
+candidate, so a lone node is still tried and a cooling node can serve after the
+others fail. It expires after one configured response-header duration, or clears
+early on a successful response. Fast errors and caller cancellation do not set
+or clear it. Nothing is persisted or shared between proxies.
+
 ### Models Do Not Need to Be on Every Node
 
 **You do not have to put the same models on every node.** A cluster where one
@@ -522,6 +531,29 @@ retryable and moves to the next advertised owner. It never fails over to an
 unknown or known-missing node. Genuine client errors such as `400` or `422` are
 not retried, because they would fail identically everywhere.
 
+Successful inference responses do not commit on headers alone. The proxy waits
+for initial body bytes (or a valid empty EOF), and pre-commit body read failures
+advance to the next existing candidate by default while retaining one request
+and workload identity. It does not re-resolve candidates or add retry rounds.
+Once the response commits, errors only terminate that stream; it is never
+replayed after bytes have reached the client.
+
+**There is no first-body deadline by default.** LM Studio can send headers before
+long-context prefill finishes, and oMLX with large MLX models on Macs can also be
+silent for several minutes. The unchanged `120s` response-header timeout
+(`--response-header-timeout` / `NVPAIR_PROXY_RESPONSE_HEADER_TIMEOUT`) does not
+limit this post-header wait.
+
+Operators may separately opt in with `--first-body-timeout` or
+`NVPAIR_PROXY_FIRST_BODY_TIMEOUT`, a positive Go duration such as `10m`. The flag
+wins over the environment; `0` explicitly disables it, including an environment
+opt-in. Invalid or negative values keep it disabled. Only an enabled first-body
+deadline can turn post-header silence into timeout-based retry and cooldown.
+The cooldown still lasts one configured response-header duration. This option
+can abandon legitimate prefill and duplicate engine work if set too short, so
+leave it off unless that trade-off is intended; see
+[Troubleshooting](troubleshooting.mdx#requests-are-silent-after-headers).
+
 #### Planning Model Placement
 
 Where you put each model decides what routing can do with it:

docs/prefix-aware-routing-spec.mdx

@@ -26,7 +26,8 @@ design below is refined by these implementation decisions:
   KV-cache residence. Failover records the serving node.
 - Affinity load includes local optimistic reservations. The eligibility check,
   promotion, and reservation use the existing scheduler lock and retain manual
-  pin precedence and failover order.
+  pin precedence and failover order within each deadline-cooldown group. A warm
+  node in cooldown cannot be promoted over a non-cooling eligible peer.
 - Plain completions require at least 1,024 bytes before prefix affinity applies.
   Ollama supplied context arrays can identify repeated context only; updated
   response token arrays are not parsed or retained.

docs/troubleshooting.mdx

@@ -154,16 +154,53 @@ runs on one selected node.
 
 A request that fails with `502` and `upstream error: net/http: timeout awaiting
 response headers` after about 120 seconds waited longer than PAIR allows for an
-engine to start answering. An engine sends nothing until generation starts, so a
-request queued behind other work, or waiting for a large model to load, can hit
-this while the engine is healthy.
+engine to send response headers. An engine such as Ollama can withhold headers
+until generation starts, so a request queued behind other work, or waiting for
+a large model to load, can hit this while the engine is healthy. This is the
+response-header deadline only, not a limit on generation or prefill after
+headers arrive.
 
 Raise the limit with the `NVPAIR_PROXY_RESPONSE_HEADER_TIMEOUT` environment
 variable, a Go duration such as `10m`, and restart PAIR. On macOS, set it for
 apps opened from the Finder with
 `launchctl setenv NVPAIR_PROXY_RESPONSE_HEADER_TIMEOUT 10m`. Every node that
 serves requests applies its own limit, so raise it on each. A longer limit also
-means a node that has stopped responding is given up on later.
+means a node that has stopped responding is given up on later. Standalone proxies
+also accept `--response-header-timeout 10m`. Header-stall cooldown remains on by
+default: a node that spent that full silent deadline is tried behind its peers
+for one header-timeout duration, never excluded.
+
+## Requests Are Silent After Headers
+
+**By default there is no first-body deadline.** LM Studio can send headers
+immediately and then spend several minutes on long-context prefill. Large MLX
+models in oMLX on Macs can also legitimately remain silent before their first
+body bytes. The `120s` response-header timeout does not cut off this wait.
+Pre-commit connection/body read failures still retry on another candidate by
+default, but silence after headers alone does not.
+
+An optional, separate first-body limit is available in all three proxies:
+`--first-body-timeout 10m` for standalone use, or
+`NVPAIR_PROXY_FIRST_BODY_TIMEOUT=10m` in PAIR's inherited environment, followed
+by a restart. On macOS, an app opened from the Finder can inherit it via
+`launchctl setenv NVPAIR_PROXY_FIRST_BODY_TIMEOUT 10m`. The flag overrides the
+environment. `0` disables it explicitly; invalid or negative values also leave
+it disabled. It does not change `--response-header-timeout` or its environment
+variable.
+
+Only enable this if you accept the trade-off: a legitimate prefill longer than
+the chosen first-body budget will be abandoned and retried, potentially
+duplicating work on another engine. An enabled first-body deadline also records
+cooldown for one configured response-header duration. Once any response body is
+committed, the stream is never retried and this first-body timer no longer
+applies.
+
+If the error says `upstream timed out before first response body byte`, this
+optional limit was enabled. Increase it or disable it rather than raising the
+unrelated response-header timeout. For a Finder-launched app, remove the opt-in
+with `launchctl unsetenv NVPAIR_PROXY_FIRST_BODY_TIMEOUT` and restart PAIR;
+standalone proxies can use `--first-body-timeout 0` even if the environment sets
+a positive value. Each node applies its own settings.
 
 ## Where the Logs Are
 

services/lmstudio-proxy/README.md

@@ -35,7 +35,8 @@ lmstudio-proxy [flags]
 | `--port` | `1234` | HTTP listen port for request forwarding |
 | `--ignore-persisted-port` | `false` | Use `--port` even when a prior runtime port was saved |
 | `--ipc` | *(empty — use stdio)* | Path to a Unix domain socket or Windows named pipe for IPC |
-| `--response-header-timeout` | `$NVPAIR_PROXY_RESPONSE_HEADER_TIMEOUT`, else `120s` | How long to wait for an upstream engine to start answering (Go duration, e.g. `10m`). An engine sends no headers until generation starts, so raise it when requests queue behind long jobs or wait on slow model loads. An invalid or non-positive value keeps the default |
+| `--response-header-timeout` | `$NVPAIR_PROXY_RESPONSE_HEADER_TIMEOUT`, else `120s` | Upstream response-header deadline only (Go duration, e.g. `10m`). Does not limit prefill after headers. An invalid or non-positive value keeps the unchanged `120s` default |
+| `--first-body-timeout` | `$NVPAIR_PROXY_FIRST_BODY_TIMEOUT`, else `0` (disabled) | Opt-in, independent wait for initial successful inference body bytes after headers (Go duration, e.g. `10m`). No post-header deadline by default. The flag overrides the environment; `0` explicitly disables it even if the environment opts in. Invalid or negative values leave it disabled |
 | `--cluster-dir` | *(empty)* | Cluster trust directory (`node.crt`/`node.key` plus trusted pins). Enables the LAN mTLS inference ingress while this node is a cluster member; empty means no ingress and no peer candidates. |
 | `--log-level` | *(`$NVPAIR_LOG_LEVEL`, else `info`)* | Initial log level: `debug`, `info`, `warn`, or `error`. Changeable at runtime with `log/set-level`. |
 | `--version` | | Print version and exit |
@@ -53,7 +54,8 @@ Node selection:
 - **Auto**: When no eligible node is explicitly selected, the proxy follows `node/set-priority` (see below), then discovered nodes in stable ID order.
 - **Priority (scheduler-driven)**: The Job Scheduler ranks the cluster least-loaded-first by pending workload plus smoothed GPU pressure and, via `nvpair-ui-broker`, pushes the ordered node list with those per-node counts to this proxy with `node/set-priority`. Auto routing sends the request to the listed node carrying the least estimated load. See [`nvpair-job-scheduler`](../nvpair-job-scheduler/README.md).
 - **Manual**: Use the `node/select` JSON-RPC method to pin traffic to a specific node. A manual pin **overrides the priority list only when that node is eligible** for the requested model.
-- **Failover**: If the selected node disappears from the discovery set, the proxy falls back to auto-select and emits a `node/selection-changed` notification. A transport error or retryable status, including a model `404` from an advertised owner with stale inventory, steps to the next eligible owner.
+- **Failover**: If the selected node disappears from the discovery set, the proxy falls back to auto-select and emits a `node/selection-changed` notification. A transport error or retryable status, including a model `404` from an advertised owner with stale inventory, steps to the next eligible owner. Successful inference responses wait for initial body bytes (or a valid empty EOF) before committing. Pre-commit body read failures retry by default within the same candidate list. There is no first-body deadline unless `--first-body-timeout` / `NVPAIR_PROXY_FIRST_BODY_TIMEOUT` explicitly opts in; only then can post-header silence trigger timeout-based retry. Once committed, a stream is never retried. Failover keeps one request/workload identity.
+- **Deadline cooldown**: A full response-header wait with no answer, or an explicitly enabled inference first-body deadline with no content, moves that candidate behind its eligible peers for one configured response-header duration. This stable demotion overrides selection, reservations, and prefix affinity but never excludes a node: a sole candidate, or a candidate after its peers fail, is still tried. State is per proxy process, in memory only; time expiry or a successful response clears it. Fast errors and caller cancellation neither set nor clear it. Header-stall cooldown remains on by default with the unchanged `120s` header deadline; first-body timing is off by default to avoid interrupting legitimate long-context prefill. See [troubleshooting](../../docs/troubleshooting.mdx#requests-are-silent-after-headers) before enabling it.
 
 ### IPC Transport
 
@@ -338,17 +340,18 @@ Semantics:
 - **Snapshot reset.** Each new snapshot replaces the pending and GPU-pressure
   baselines and clears the reservations. GPU pressure is clamped to the
   scheduler's 0–3 range.
-- **Eligible manual pin wins.** An active `node/select` pin takes precedence when
-  it is in the request's owner set. An ineligible pin is ignored for that request,
+- **Eligible manual pin wins unless cooling.** An active `node/select` pin
+  takes precedence when it is in the request's owner set and is not behind
+  peers in deadline cooldown. An ineligible pin is ignored for that request,
   so automatic reservations still apply among eligible owners. Clearing the pin
   (`node/select` with an empty `id`) activates the most-recently-set list. Setting
   a priority list does **not** emit `node/selection-changed` (the manual selection
   is unchanged).
 - **Unknown IDs are ignored.** IDs not currently in discovery are kept in the
   stored list (a node may appear later) but contribute nothing until discovered.
 - **Eligible unlisted nodes are a lowest-priority fallback.** An advertised owner
-  absent from the list stays routable, but only after every listed owner —
-  ordered among themselves by the default stable ID sort. This ensures an
+  absent from the list stays routable, but only after every listed owner in
+  the same cooldown group — ordered among themselves by the default stable ID sort. This ensures an
   eligible manually-added node the scheduler never saw is never stranded.
 - **Empty list reverts to default.** `{"nodes":[]}` clears the scheduler's
   influence and returns the proxy to its default auto ordering (eligible