Repository navigation
Evict the longest-idle session instead of refusing at the connection limit (v1.9.7) - #29
Merged
Merged
Conversation
…limit (v1.9.7) ChatGPT opens a new MCP session for every tool call and never sends DELETE: initialize, notifications/initialized, a GET it drops after a millisecond, one tools/call. Each of those sessions counted as used, so it held its slot for the full 10-minute idle timeout, and an intensive ChatGPT user ran into "429 Too many concurrent connections" after 20 calls. The 30-second timeout for unused sessions never fired, because the initialized notification and the GET already marked a session used. The pool now limits how many sessions of a token may work at once, not how many it has walked away from. When it is full, the session of the same token that has been idle longest and has no POST in flight is closed to make room. A client returning to it gets 404 and initializes again, as the MCP spec requires. Only when all 20 are mid-request does the client get 429. The used flag and the 30-second timeout are gone. Visibility, which is why this went unnoticed twice: - "Session initialized", "Session evicted" and the per-token 429 log the token fingerprint, so a burst can be attributed to a tenant. - Rejections by the connection limit, the pre-auth limit and the OAuth limit are counted in mcp_rate_limit_exceeded_total; evictions in the new mcp_sessions_evicted_total. - An initialize request that ends without a session gets a log line. - Tool call log lines carry the session fingerprint, not the raw ID. tests/sessionEviction.spec.ts replays the incident: 29 sessions over two minutes, each used for one tool call. It failed before this change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
ChatGPT opens a new MCP session for every tool call and never sends
DELETE(initialize →notifications/initialized→ a GET it drops after ~1 ms → onetools/call). Every such session counted as used, so it kept its connection slot for the full 10-minute idle timeout. An intensive ChatGPT user hit429 Too many concurrent connectionsafter 20 calls in 10 minutes.This is the third time the pattern showed up. The earlier fixes treated the symptom: raising the limit from 5 to 20 (91ef335), then a 30-second timeout for unused sessions (f62351b). That timeout never fired for ChatGPT, because the
initializednotification and the GET already marked a session as used. Production logs for the incident: 72 sessions, 38 tool calls on 38 different sessions, 65 of 65 idle closes withused=true.Change
404and must re-initialize (MCP spec, Session Management). Only when all 20 sessions are mid-request does the client get429. Other tenants' sessions are never touched.usedflag andunusedSessionTimeoutMsare removed. One idle timeout (10 min) remains, and the sweeper skips sessions with a request in flight.Session initialized,Session evictedand the per-token429.mcp_rate_limit_exceeded_total{type}now also countsconnections,pre_authandoauth.mcp_sessions_evicted_total.The per-IP and per-token request limits are unchanged.
Tests
tests/sessionEviction.spec.tsreplays the incident: 29 sessions over two minutes, each with the full ChatGPT sequence. It was red before this change. It also covers 404 on an evicted session, the choice of the longest-idle session, the in-flight guard (429 only when all sessions are busy), tenant isolation, and the logs and metrics.tests/connectionSlots.spec.ts: the unused-timeout tests are removed. The DELETE test now asserts that no eviction was needed.npm test(566),npm run lintandnpm run typecheckpass.After merge
Build and scan the image locally, then tag
release-v1.9.7. The changelog should say thatmcp_rate_limit_exceeded_totalhas newtypevalues and thatmcp_sessions_evicted_totalis new, in case dashboards or alerts filter on these.🤖 Generated with Claude Code