Repository navigation
Make the proxy header timeout configurable and take action timeouts from manifests - #30
Merged
Merged
Conversation
An engine sends no headers until generation starts, so a request queued behind other work, or waiting on a slow model load, failed through PAIR with 502 "timeout awaiting response headers" after a hard-coded 120 s, while the same request sent to the engine directly succeeded. Each proxy now takes --response-header-timeout, then NVPAIR_PROXY_RESPONSE_HEADER_TIMEOUT, then the 120 s default. The proxies inherit the broker's environment, so on macOS `launchctl setenv` reaches an app opened from the Finder. An invalid or non-positive value keeps the default. The default is unchanged: these proxies have no retry for a candidate that has stopped answering, so a longer wait would make a frozen node more expensive. Adapted from NVIDIA#83 (issue NVIDIA#57) for this fork's three proxies. Co-authored-by: Mallikh Kalkere <mkalkere@users.noreply.github.com> Signed-off-by: Jared Ping <13906915+JaredP94@users.noreply.github.com>
…nifest The engine manager gave a 10-minute response-header wait to Ollama's run_model and to any load_model by name, and 30 s to everything else, so LM Studio's and oMLX's chat actions failed on a long prefill or a cold load. An HTTP action now declares timeout_s in its manifest; the bundled run_model, load_model and chat actions declare 600, and an action that declares none keeps 30 s. Validation rejects a negative value or one on a non-HTTP action. Adapted from NVIDIA#93 (issue NVIDIA#25). Co-authored-by: Aaron K. Clark <cryptojones@owasp.org> Signed-off-by: Jared Ping <13906915+JaredP94@users.noreply.github.com>
…r wait Review follow-up. The local oMLX and LM Studio load actions now wait up to 600 s for headers, but a load requested from a cluster peer still used the 30 s client unless the engine was Ollama, and cancelling it propagates to the peer's in-flight action. Every remote load now uses the readiness client; the peer's own action timeout still bounds it. Signed-off-by: Jared Ping <13906915+JaredP94@users.noreply.github.com>
JaredP94
force-pushed
the
feat/configurable-timeouts
branch
from
November 2, 2026 18:48
1872df9 to
0a6b9a8
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
ollama-proxy,lmstudio-proxy,omlx-proxy). An engine sends no headers until generation starts, so a request queued behind other work or waiting on a slow model load failed with502 … timeout awaiting response headersafter a hard-coded 120 s. Each proxy now takes--response-header-timeout, thenNVPAIR_PROXY_RESPONSE_HEADER_TIMEOUT, then 120 s. Invalid/non-positive values keep the default. The default is unchanged on purpose: these proxies don't yet retry a candidate that stopped answering, so a longer wait would make a frozen node more expensive. Troubleshooting docs cover it (launchctl setenvon macOS). Adapted from Make proxy upstream response-header timeout configurable NVIDIA/Personal-AI-Router#83 ([Bug]: ollama-proxy: undocumented 120 s upstream response-header timeout turns a queued or cold-loading request into a 502 NVIDIA/Personal-AI-Router#57).nvpair-engine-manager). The 10-minute header wait was given to Ollamarun_modeland anyload_modelby name; LM Studio's and oMLX'schatgot 30 s. HTTP actions now declaretimeout_s; the bundledrun_model,load_modelandchatactions declare 600. Adapted from Select action response-header timeouts from the manifest NVIDIA/Personal-AI-Router#93 ([Bug]: Action HTTP response-header timeout is selected by a hardcoded engine name, so only Ollama gets the long client NVIDIA/Personal-AI-Router#25).Validation
go test ./...in all four modules, with resolver, transport and per-action tests. Confirmed broker and desktop spawn children with the inherited environment.Risk
Low. Proxies 1.4.1 / 1.4.1 / 1.4.2, engine-manager 1.4.0.
Checklist
git commit -s), certifying the Developer Certificate of Origin.services/versions.json, and described user-visible changes above so they reach the release notes.Part of the upstream issue/PR review. Each branch is independent off
main; they all bumpservices/versions.json, so the second one merged will need a trivial conflict resolution there.