[JaredP94]

Description

A node can accept connections without returning response headers, making every request pay the full upstream deadline before failover. All three proxies now remember that measurement and move the stalled candidate behind its eligible peers for one configured response-header duration. Time expiry or a successful response restores normal ordering; the node is never excluded.

Successful inference waits for initial response-body bytes (or a valid empty EOF) before committing, so a pre-commit drop can use the existing failover list by default. Retries keep one request/workload identity and never replay a committed stream. The response-header default remains 120 seconds, and there is no first-body deadline by default. Post-header silence triggers timeout-based retry only when explicitly enabled with --first-body-timeout or NVPAIR_PROXY_FIRST_BODY_TIMEOUT.

Addresses NVIDIA#6. Adapted from NVIDIA#144 and the pre-commit retry portion of NVIDIA/Personal-AI-Router@e0f75dd.

Scope

• Identical stall/cooldown helpers and regression tests in ollama-proxy, lmstudio-proxy, and omlx-proxy.
• Stable demotion after candidate resolution; reservations, manual selection, and prefix affinity cannot promote a cooling candidate over a ready peer. Sole/all-cooling candidates remain usable.
• Header-stall measurements begin after the request is written, excluding dial/TLS time and caller cancellation. Fast errors do not create a cooldown. Header-stall cooldown and pre-commit error retry remain enabled by default.
• Independent, opt-in --first-body-timeout / NVPAIR_PROXY_FIRST_BODY_TIMEOUT accepts a positive Go duration. Default 0 disables it; the flag overrides the environment, and explicit 0 disables an environment opt-in. Invalid/negative values leave it disabled. Only an enabled first-body timer can turn post-header silence into retry/cooldown; cooldown still lasts one response-header duration.
• No retry rounds, discovery watcher, or job-state/JSON-RPC changes. No change to the response-header timeout or its configuration.
• Component READMEs, repository README, architecture, affinity, and troubleshooting documentation updated to distinguish both settings.
• Version decision: retain the existing unreleased patch bumps relative to v0.8.0: Ollama and LM Studio 1.4.0 → 1.4.1, oMLX 1.4.1 → 1.4.2. No additional bump is needed under the implementation brief's fork practice. No changelog edits.

Validation

On macOS arm64, in each of the three touched Go modules:

test -z "$(gofmt -l *.go)"
go vet ./...
GOOS=linux go vet ./...
GOOS=windows go vet ./...
go test -count=1 ./...
go test -race -count=1 ./...
go test -count=3 -run '^(TestStalledCandidateSecondRequest|TestDefaultFirstBodyWaitIsUnbounded)$' ./...

All passed after the review changes. At repository root, git diff --check passed and node scripts/spdx-headers.mjs reported 0 missing.

Regression evidence in every proxy:

• Frozen httptest upstreams cover accepting without headers under default configuration, plus headers-without-body with an explicit flag opt-in and an environment opt-in. A second request reaches only the healthy peer, which must receive the complete original request body on both requests.
• Running workload notifications must name the healthy serving node before terminal completion, under the same workload identity. Correct terminal placement alone no longer satisfies the test.
• TestDefaultFirstBodyWaitIsUnbounded lets a healthy upstream remain silent after headers for three times the shortened test header budget. It completes without failover or cooldown. Before the fix, this new regression failed in all three proxies by dispatching duplicate work to the fallback.
• Temporarily truncating forwarded request bodies made the strengthened body assertions fail in every proxy. Temporarily suppressing only the running re-point notification, while preserving correct terminal placement, made the running-event assertions fail in every proxy. Both controls were restored before final verification.
• Resolver tests cover the disabled default, flag/env opt-in, precedence, explicit zero, invalid/negative values, and independence from the header deadline. Existing coverage retains pre-commit drop retry with the first-body timer disabled and bounded final-candidate failure with the timer enabled.
• Initial implementation negative controls separately proved that disabling cooldown paid the frozen-node deadline twice, and removing the first-body gate broke pre-commit drop retry and enabled body-stall bounds.
• Expiry, success recovery, independent proxy state, stable ordering, selected/unlisted candidates, reservations/affinity, sole/all-cooling candidates, caller cancellation, empty successful bodies, and committed-stream no-retry coverage remain passing.
• A repeated run initially outlasted the regression's short cooldown before the healthy response finished. The test now leaves a three-second header/cooldown window while the first-body opt-ins remain independently set to one second; full suites, race suites, and three repeated runs passed afterward.

No desktop files changed. No real engine installation, live-cluster, or cross-process tests were run. Linux/Windows checks are static vet only, not runtime certification. The initial secondary-model review attempt timed out without a report.

Risk

• No first-body deadline by default: LM Studio may send headers immediately and then spend several minutes on legitimate long-context prefill; large MLX models in oMLX on Macs can also be silent before content. The unchanged 120-second response-header timeout does not limit that post-header wait. A truly stalled upstream that has already sent headers can therefore keep waiting until the caller cancels unless the operator opts in.
• Opt-in trade-off: a positive --first-body-timeout / NVPAIR_PROXY_FIRST_BODY_TIMEOUT can abandon legitimate prefill and duplicate work on a second node if set too short. Leave it disabled unless that trade-off is intended; choose a budget appropriate for long contexts, cold loads, and queues. The flag overrides the environment, and --first-body-timeout 0 explicitly disables it. Header and enabled first-body waits are separate and can both be incurred before failover.
• Engines such as Ollama can withhold headers during legitimate work. Their existing response-header deadline is unchanged; increase --response-header-timeout / NVPAIR_PROXY_RESPONSE_HEADER_TIMEOUT if necessary, independently of the optional first-body setting.
• Demotion is process-local; proxies may temporarily disagree about a peer. No state is persisted or exchanged.
• As with existing pre-commit failover, the abandoned engine may have started work. This does not promise exactly-once engine execution; it preserves one visible workload and never splices a replacement into a committed client stream.

Checklist

• I have read the Contributing Guidelines.
• Every commit is signed off (git commit -s), certifying the Developer Certificate of Origin.
• New or existing tests cover the change.
• Relevant documentation is updated.
• I checked the diff, changed filenames, and commit messages for credentials, private data, internal URLs, internal issue identifiers, and generated artifacts.
• I recorded the validation commands and results above.
• Affected component versions already carry the required unreleased patch bumps; the version decision and user-visible changes are described above for release notes.