Repository navigation
fix(mcp-gateway): re-handshake a pooled backend whose server lost its session - #16382
Conversation
… session When the command the gateway spawns is a pooling multiplexer's thin client, the multiplexer can replace the real server behind the pipe (retired, died) and start the replacement cold, while the client's own connection stays up. The gateway sent initialize once for that backend and answers every later stub from its cache, so nothing handshakes the new process: a Python MCP server then refuses every request with -32602 "Invalid request parameters" and empty data until the gateway restarts. A standalone kiro-cli run works because it is a fresh connection with a fresh handshake. On exactly that refusal, for a request safe to send twice, the backend now re-sends its cached initialize, notifications/initialized and the request in one write, once. The initialize reply is swallowed; the stub sees only the retry's answer under its own id. Any other error, a second refusal, or a detached stub is delivered unchanged.
First Principles Review (Opus 5.5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Check before merge: the supplied patch has about 170 files outside What this change shipsInventory (5 items) — 5 justifiedIntent: keep tools from a multiplexer-pooled MCP server working from dashboard chats and subagents after the multiplexer replaces the server process behind them. This is a FIX.
[FIRST-PRINCIPLES-REVIEWED] e7a0e6f Description read[DESCRIPTION-READ] 49c165e2a7b74c075ae44d3601326f1f4e8d0a9a237d41ea93b251b4235e3ca2 That sha256 names the PR description this verdict read. Recompute it with |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Check that no other MCP SDK sends [DESIGN-REVIEWED] e7a0e6f Description read[DESCRIPTION-READ] 83eaca6d7d9b7f06d3421d7f804d9ea6d604c65d5d82c42413170224fd322a8e That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findings (all downgraded on adjudication)GPT 6.1 flagged blocking issues on This comment is updated in place on each push. Review detailsBLOCKING -- website/src/components/markdown/MarkdownTable.tsx:150 -- Widen adds a third action to the table’s button row Adjudication (Opus 5.5) — is blocking on each finding proportionate?F1: LOW. A third button in a hover-revealed table action row is a visible layout and UX issue, with no data, security or correctness effect.
False positive or not applicable? A repository writer can comment: |
UX Review (Opus 5.5) — ✅ PASSUX-level review of UX-Verdict: PASS Nothing to check. [UX-REVIEWED] e7a0e6f Description read[DESCRIPTION-READ] dbf1c77aa450abdb775a305a11e994ccaefac56bba7eaa3ef9818666d972c541 That sha256 names everything this verdict read: the PR description, plus the 3 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
Problem / Motivation
Goal: MCP tools served through a pooling multiplexer keep working from dashboard chats and subagents after the multiplexer replaces the server process behind them.
A user reported that every tool of every multiplexer-pooled Python MCP server fails from the dashboard and from subagents with
-32602: Invalid request parameters(""), while the same agent, tool and arguments succeed in a standalonekiro-cli chat --agent kirocrewrun. It reads like the gateway dropping the tool arguments. It is not: that frame is the Python MCP SDK refusing a request on a session that never receivedinitialize+notifications/initialized(mcp/shared/session.pyreports theReceived request before initialization was completeerror as INVALID_PARAMS with emptydata). An emptyargumentsobject gets a tool-level validation error withisError: trueinstead, not -32602.Why it matters
Once the multiplexer recycles its server instance (credential rotation, idle reap, a crash), every later call on that pooled backend fails until the gateway restarts. The only workaround is to leave Kiro Crew and run the call in a standalone kiro-cli session.
Not a goal
initialize,resources/subscribe/unsubscribe, or any method outside the retry set.What changed (motivation -> approach -> change)
Symptom:
-32602 Invalid request parameterswith empty data on every call, only under the gateway.Root cause: the configured command is the multiplexer's thin client. The client keeps its connection to the multiplexer's daemon while the daemon replaces the real server and starts the replacement cold, sending it nothing before the next request. The gateway handshakes each backend once (
_handle_initialize) and answers every later stub from_init_result, and kiro-cli sendsinitializeonce per session, so nobody handshakes the new process. A standalone kiro-cli run is a fresh connection with a fresh handshake, which is why it works.Change, in
src/kiro_crew/mcp_gateway/backend.py:initializeframe is kept onBackend._upstream_init_frame(both send sites:_handle_initialize,prime_initialize).forward_from_stubkeeps the frame as it went upstream (identity blocks included) on_PendingRequest.retry_frame, only for methods in_REHANDSHAKE_RETRY_METHODS(tools/call, tools/list, prompts/list, prompts/get, resources/list, resources/read, resources/templates/list, completion/complete). The refusal comes before the SDK dispatches anything, so each is safe to send twice._route_backend_line: when a response matches_is_lost_session_errorand its pending has aretry_frame,_retry_after_rehandshakewritesinitialize(new gateway id, reply swallowed under_REHANDSHAKE_STUB_SENTINEL),notifications/initializedand the request (new id, same pending fields,retry_frame=None) in ONE write via the new_write_json_lines. One write keeps the retry behind the handshake and keeps a co-tenant's request out of the gap betweeninitializeandinitialized, where a Python server refuses everything. The retry is never retried: a second refusal reaches the stub unchanged.ClientRequestvalidation (non-objectarguments, malformed_meta) gets the same frame. Such a request now costs one extra handshake and one retry before its error reaches the caller unchanged. A secondinitializeon a healthy Python server is harmless: it moves the session to initializing, theinitializedin the same write moves it back, and later calls work (see Manual verification). The kiro-cli client buildsargumentsas an object, so this path should be rare.docs/architecture/mcp.mdTroubleshooting gets the symptom and the mechanism.Backwards compatibility
Compatible: only a response that already fails the call is affected, and only for the one frame the Python SDK sends for an uninitialized session. Every other answer and every other method takes the existing path.
Removes nothing: the diff adds a recovery path and two fields; no key, validator, record or public function is removed or tightened.
Tests
test/test_mcp_gateway_rehandshake.py(new):test_lost_session_is_handshaken_again_and_the_call_retried_once: one write with the original upstreaminitialize,initializedand the same request under a new id; the refusal never reaches the stub; theinitializereply is swallowed; the retry's answer reaches the stub under its own id; no pending is left.test_a_retry_refused_again_is_delivered_not_retried: no loop.test_other_errors_are_delivered_untouched(3 cases): a tool-filter "Tool X not found" -32602, the same message with non-empty data, and a different code.test_a_method_outside_the_retry_set_is_delivered_untouchedtest_no_retry_for_a_stub_that_has_detachedtest_the_retry_carries_the_injected_identity_blocksThe two positive tests fail on origin/main; the negative ones pass on both.
test/test_mcp_gateway*.py(49 files) pass locally excepttest_mcp_gateway_adopted_daemon_repair.py::test_a_stale_survivor_is_repaired_end_to_end, which fails the same way on origin/main in this sandbox (unshare EPERM).Manual verification
End to end with real processes: a gateway
Backendfromspawn_backendrunning a real multiplexer thin client, its real daemon, and a Python FastMCP server (mcp 1.17.0) with one string-argument tool. The server process is killed after the first call; the daemon starts a replacement and the client connection stays up.Healthy server, malformed request (
argumentsa string), then exactly what the gateway sends (initialize+initialized+ the same request in one write), then a normal call:Raw-frame probe against the same server:
argumentsset to{},nullor left out givesisError: true(tool validation), never -32602. -32602 with empty data comes back only for a request beforeinitialized, a non-objectarguments, or a malformed_meta.Related Issues
None filed. Reported by a user; root cause reproduced as described above.
Pattern harvest
Not generalizable: the gateway caches one handshake per backend on purpose. The defect is a peer that loses its session behind a live pipe, and recognising that needs the peer's own error frame. No rule can find that statically.
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)Contribution License Agreement