Switch to httpx2 and reuse connections for streaming requests by QHarshil · Pull Request #1282 · vllm-project/guidellm

QHarshil · 2026-10-10T04:22:15Z

Summary

Streaming requests never reuse a connection. The stream loop stops at [DONE] before the response ends, so the connection is closed, and at high request rates the client runs out of local ports. At 800 req/s for 60 s against a local server, 12,078 requests failed with ConnectError on main. This reads the rest of each HTTP/1.1 response in the background so its connection goes back to the pool, and switches to httpx2, whose pool handles that reuse without the stall httpx's pool has. Fixes cause 2 of #1238 and closes #1300.

Details

Replace httpx with httpx2 in the dependencies, source, and tests. Tests mock requests with pytest-httpx2.
After the final event of an HTTP/1.1 stream, a background task reads the rest of the response for at most 1 s and closes it. The request returns at once, and process_shutdown waits for these tasks. HTTP/2 streams stop at the final event.
Connections come from the client's shared pool, so HTTP/2 multiplexing and environment proxies work as on main.
httpx2 verifies TLS certificates against the operating system's trust store, or SSL_CERT_FILE / SSL_CERT_DIR when set. httpx used certifi's bundle.

Test Plan

tox -e lint-check, type-check, test-integration: pass.
tox -e test-unit, test-e2e: only the libtorchcodec failures in test_audio.py and test_encoders.py and the test_realtime_ws_e2e error, same as main.
Reading inline, not waiting at shutdown, reading HTTP/2 to the end, never reading to the end, removing the 1 s limit, or setting a custom transport each fails a test.
Test server from Reported TTFT includes time spent inside GuideLLM #1238 on macOS, 10 workers, 20 ms TTFT, 64 output tokens, two runs each:

	main	this PR
errored requests, 800 req/s for 60 s	12,078	0
TCP connections, 1024 streams	15,983, 16,009	1,028, 1,028
wait before send p99 (ms), 1024 streams	29, 26	3.3, 3.0
requests/s, 1024 streams	744, 746	739, 744
TTFT p99 (ms), 1024 streams	36, 42	91, 92
server's own TTFT p99 (ms), 1024 streams	20.6, 20.6	25.1, 25.6

With reuse, every request in a wave of fixed-length streams is sent within about a millisecond, so first tokens reach the server and each worker together and TTFT p99 rises at high concurrency. On main, opening a new connection per request spreads each wave out.
HTTP/2 over TLS, four runs each: TTFT p50 at 256 streams was 24 to 47 ms on main and 24 to 33 ms with this PR. At 1024 streams both are slow and vary, 86 to 636 ms on main and 67 to 258 ms with this PR.

Related Issues

Part of Reported TTFT includes time spent inside GuideLLM #1238
Closes Drop httpx in favor of httpx2 #1300

"I certify that all code in this PR is my own, except as noted below."

Use of AI

Includes code generated or substantially modified by an AI agent
Includes tests generated or substantially modified by an AI agent

NOTE: the Generated-by or Assisted-by trailers should be used in git commit messages when code or tests were generated or substantially modified by an AI agent, as described in the project's DEVELOPING.md file.

git log

commit 99db2f6
Author: QHarshil harshil_c@hotmail.com
Date: Fri Oct 9 21:21:57 2026 -0700

Reuse connections for streaming requests

Read HTTP/1.1 streams to the end so their connection returns to the
pool, and give each in-flight request its own pool.

Signed-off-by: QHarshil <harshil_c@hotmail.com>

commit 8b7ebf5
Author: QHarshil harshil_c@hotmail.com
Date: Fri Oct 9 21:48:25 2026 -0700

Honor environment proxies with per-request pools

A custom transport turns off httpx's environment proxy handling, so mount a
per-request pool for each proxy from the environment.

Signed-off-by: QHarshil <harshil_c@hotmail.com>

commit d87fa2c
Author: QHarshil harshil_c@hotmail.com
Date: Wed Oct 14 12:10:35 2026 -0700

Read streaming responses to the end in the background

Drop the per-request pools. After the final event on HTTP/1.1, a task reads
the rest of the response so the request returns at once, and
process_shutdown waits for those tasks.

Signed-off-by: QHarshil <harshil_c@hotmail.com>

commit 14036da
Author: QHarshil harshil_c@hotmail.com
Date: Wed Oct 14 12:10:35 2026 -0700

Switch from httpx to httpx2

httpcore2 rewrites the pool's request assignment, which removes the stall a
shared pool hit once connections were reused. Tests mock with pytest-httpx2.

Signed-off-by: QHarshil <harshil_c@hotmail.com>

Signed-off-by: QHarshil harshil_c@hotmail.com

Read HTTP/1.1 streams to the end so their connection returns to the pool, and give each in-flight request its own pool. Signed-off-by: QHarshil <harshil_c@hotmail.com>

chatgpt-codex-connector · 2026-10-10T04:22:24Z

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-10T04:27:33.372712Z	`99db2f6`	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 99db2f674b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

A custom transport turns off httpx's environment proxy handling, so mount a per-request pool for each proxy from the environment. Signed-off-by: QHarshil <harshil_c@hotmail.com>

sjmonson

I see some logic in ensuring we properly close streams, but the per-request transports doesn't make sense.

Drop the per-request pools. After the final event on HTTP/1.1, a task reads the rest of the response so the request returns at once, and process_shutdown waits for those tasks. Signed-off-by: QHarshil <harshil_c@hotmail.com>

httpcore2 rewrites the pool's request assignment, which removes the stall a shared pool hit once connections were reused. Tests mock with pytest-httpx2. Signed-off-by: QHarshil <harshil_c@hotmail.com>

mergify · 2026-10-14T21:46:39Z

Received a single approval. Will automatically queue for merge once two maintainers approve, all change requests are resolved, and DCO passes.

mergify · 2026-10-14T21:46:41Z

Queued — the merge queue status continues in this comment ↓.

mergify · 2026-10-15T14:41:40Z

jaredoconnell

Looks good to me. It even performs better on my mock server.

main moved the mock server tests from httpx to httpx2 in vllm-project#1282, so the two new tests no longer resolved httpx.AsyncClient. Assisted-by: Claude Code claude-opus-5-5 Signed-off-by: MinJun Choi <mj04300017@gmail.com>

## Summary Fixes `max_concurrent_requests` for streaming requests. Right now the slot is released as soon as the handler returns its `ResponseStream`, before any tokens are generated, so streaming requests were never limited. This holds the slot until the stream actually finishes. ## Details - [x] In `_run_generation`, a streaming response's `streaming_fn` is wrapped so the slot is released when the stream ends, including when the client disconnects partway through. Non-streaming requests work the same as before. - [x] Two new tests: overlapping streams now run one at a time, and a stream the client abandons still frees its slot so the next request doesn't hang. ## Test Plan - `lint-check`, `type-check`, `test-unit` (3,658 passed), `test-integration` (32 passed) and `test-e2e` (14 passed) all pass. - The streaming test fails on `main`, and the disconnect test fails if the slot is only released after a stream completes normally. - Same repro as the issue, with this change: the second streaming request now waits for the slot (0.51s / 1.01s), same as non-streaming. ## Related Issues - Resolves #1261 --- - [x] "I certify that all code in this PR is my own, except as noted below." ## Use of AI - [x] Includes code generated or substantially modified by an AI agent - [x] Includes tests generated or substantially modified by an AI agent --- # git log commit 3cd601e Author: MinJun Choi <mj04300017@gmail.com> Date: Thu Oct 8 13:39:40 2026 +0900 Hold the mock server's concurrency slot for the whole stream max_concurrent_requests is meant to cap in-flight generation requests, but for streaming requests it didn't cap anything. The semaphore was only held while the handler ran, and a streaming handler returns its ResponseStream right away. Sanic runs the actual generation afterwards, so the slot was already free by the time the first token went out. Now the slot is released when the stream finishes instead, including when the client disconnects partway through. Non-streaming requests behave as before. Assisted-by: Claude Code claude-opus-5-5 Signed-off-by: MinJun Choi <mj04300017@gmail.com> commit 416e962 Author: MinJun Choi <mj04300017@gmail.com> Date: Thu Oct 8 13:39:40 2026 +0900 Test max_concurrent_requests with streaming requests The existing test only covered non-streaming requests, which is how the streaming gap went unnoticed. Add one test that overlapping streams run one at a time, and one that a stream the client abandons still frees its slot, so the next request doesn't hang. Assisted-by: Claude Code claude-opus-5-5 Signed-off-by: MinJun Choi <mj04300017@gmail.com> commit 9b3167e Author: MinJun Choi <mj04300017@gmail.com> Date: Fri Oct 9 22:31:23 2026 +0900 Compare stream finish times in the streaming concurrency test The test passed when the slower of two overlapping streams took at least 0.3s. Without the fix the streams run together and the slower one took about 0.26s, so a busy CI runner could push it past the bound and let the test pass on a broken server. Check the gap between the two finish times instead. Run together, they finish within about a millisecond of each other; serialized, the second finishes about a ttft later. Over ten runs each, the gap was 0.000-0.001s without the fix and 0.205-0.220s with it, and a slower runner delays both streams without closing the gap. Assisted-by: Claude Code claude-opus-5-5 Signed-off-by: MinJun Choi <mj04300017@gmail.com> commit 75e83c9 Author: MinJun Choi <mj04300017@gmail.com> Date: Fri Oct 16 11:11:17 2026 +0900 Use httpx2 in the streaming concurrency tests main moved the mock server tests from httpx to httpx2 in #1282, so the two new tests no longer resolved httpx.AsyncClient. Assisted-by: Claude Code claude-opus-5-5 Signed-off-by: MinJun Choi <mj04300017@gmail.com> --------- Assisted-by: Claude Code claude-opus-5-5 Signed-off-by: MinJun Choi <mj04300017@gmail.com>

## Summary Fixes #1432. The Prometheus collector added in #1250 imports `httpx`, which is no longer a declared dependency after #1282 moved the codebase to `httpx2`. On a pip install of `main`, every `guidellm run` fails with `ModuleNotFoundError: No module named 'httpx'`. ## Related Issues - Resolves #1432 ## Details - `src/guidellm/collectors/prometheus.py`: `httpx` → `httpx2` (`AsyncClient`, `AsyncBaseTransport`, `HTTPError`, `InvalidURL`). - `tests/unit/collectors/test_prometheus.py`: `httpx` → `httpx2` (`Request`, `Response`, `MockTransport`, `ConnectError`). `httpx2` exports all of these with the same API, and `InvalidURL` is still a separate exception from `HTTPError`, so the `except` clause behaves the same. No functional change. ## Test Plan In a venv with `httpx` uninstalled (`pip show httpx` → not found), WSL2 / Python 3.14.4: - `python -m pytest tests/unit/collectors/test_prometheus.py`: 20 passed - `guidellm run` against `guidellm mock-server` with synthetic data (concurrent@10, 20 requests): 20/20 completed, 0 errored. Before this change, the same command failed at import with `ModuleNotFoundError: No module named 'httpx'`. ## Use of AI The change was drafted with Claude's help. I applied it and ran the tests above locally myself. --- - [x] "I certify that all code in this PR is my own, except as noted below." - [x] Includes code generated or substantially modified by an AI agent - [ ] Includes tests generated or substantially modified by an AI agent --------- Assisted-by: Claude Signed-off-by: Swetha Kommuri <swethakommuri10@gmail.com>

Reuse connections for streaming requests

99db2f6

Read HTTP/1.1 streams to the end so their connection returns to the pool, and give each in-flight request its own pool. Signed-off-by: QHarshil <harshil_c@hotmail.com>

QHarshil requested review from dbutenhof, dfeddema, jaredoconnell and sjmonson as code owners October 10, 2026 04:22

chatgpt-codex-connector Bot reviewed Oct 10, 2026

View reviewed changes

Comment thread src/guidellm/backends/openai/http.py Resolved

Honor environment proxies with per-request pools

8b7ebf5

A custom transport turns off httpx's environment proxy handling, so mount a per-request pool for each proxy from the environment. Signed-off-by: QHarshil <harshil_c@hotmail.com>

sjmonson requested changes Oct 12, 2026

View reviewed changes

Comment thread src/guidellm/backends/openai/http.py Resolved

Comment thread src/guidellm/backends/openai/http.py Resolved

QHarshil added 2 commits October 14, 2026 12:10

Read streaming responses to the end in the background

d87fa2c

Drop the per-request pools. After the final event on HTTP/1.1, a task reads the rest of the response so the request returns at once, and process_shutdown waits for those tasks. Signed-off-by: QHarshil <harshil_c@hotmail.com>

Switch from httpx to httpx2

14036da

httpcore2 rewrites the pool's request assignment, which removes the stall a shared pool hit once connections were reused. Tests mock with pytest-httpx2. Signed-off-by: QHarshil <harshil_c@hotmail.com>

QHarshil changed the title ~~Reuse connections for streaming requests~~ Switch to httpx2 and reuse connections for streaming requests Oct 14, 2026

sjmonson approved these changes Oct 14, 2026

View reviewed changes

dbutenhof approved these changes Oct 15, 2026

View reviewed changes

mergify Bot added the queued label Oct 15, 2026

mergify Bot mentioned this pull request Oct 15, 2026

merge queue: checking #1282 on main (89e440e) #1310

Closed

17 of 27 tasks

jaredoconnell approved these changes Oct 15, 2026

View reviewed changes

mergify Bot merged commit 79c8078 into vllm-project:main Oct 15, 2026
10 of 11 checks passed

mergify Bot removed the queued label Oct 15, 2026

theminjunchoi mentioned this pull request Oct 16, 2026

Hold the mock server's concurrency slot for the whole stream #1262

Merged

5 tasks done

This was referenced Oct 22, 2026

[Bug]: guidellm run fails on pip install of main — ModuleNotFoundError: No module named 'httpx' (regression from #1250) #1432

Closed

fix(collectors): use httpx2 in Prometheus collector #1433

Merged

Navigation Menu

Repository navigation

Switch to httpx2 and reuse connections for streaming requests - #1282

Switch to httpx2 and reuse connections for streaming requests#1282
mergify[bot] merged 4 commits into
vllm-project:mainfrom
QHarshil:fix/streaming-connection-reuse

QHarshil commented Oct 10, 2026 •

edited

Loading

Uh oh!

chatgpt-codex-connector Bot commented Oct 10, 2026 •

edited

Loading

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

Uh oh!

sjmonson left a comment

Uh oh!

Uh oh!

Uh oh!

mergify Bot commented Oct 14, 2026

Uh oh!

mergify Bot commented Oct 14, 2026 •

edited

Loading

Uh oh!

mergify Bot commented Oct 15, 2026 •

edited

Loading

Uh oh!

jaredoconnell left a comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

QHarshil commented Oct 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Details

Test Plan

Related Issues

Use of AI

git log

Uh oh!

chatgpt-codex-connector Bot commented Oct 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codex Review Summary

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

sjmonson left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

mergify Bot commented Oct 14, 2026

Uh oh!

mergify Bot commented Oct 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mergify Bot commented Oct 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merge Queue Status

Uh oh!

jaredoconnell left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

QHarshil commented Oct 10, 2026 •

edited

Loading

chatgpt-codex-connector Bot commented Oct 10, 2026 •

edited

Loading

mergify Bot commented Oct 14, 2026 •

edited

Loading

mergify Bot commented Oct 15, 2026 •

edited

Loading