Skip to content

fix(mcp-gateway): re-handshake a pooled backend whose server lost its session - #16382

Merged
iamwhatever merged 2 commits into
mainfrom
fix/mcp-gateway-rehandshake-lost-session
Oct 3, 2026
Merged

iamwhatever merged 2 commits into
mainfrom
fix/mcp-gateway-rehandshake-lost-session

Conversation

@chenmingwei23

@chenmingwei23 chenmingwei23 commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: MCP tools served through a pooling multiplexer keep working from dashboard chats and subagents after the multiplexer replaces the server process behind them.

A user reported that every tool of every multiplexer-pooled Python MCP server fails from the dashboard and from subagents with -32602: Invalid request parameters(""), while the same agent, tool and arguments succeed in a standalone kiro-cli chat --agent kirocrew run. It reads like the gateway dropping the tool arguments. It is not: that frame is the Python MCP SDK refusing a request on a session that never received initialize + notifications/initialized (mcp/shared/session.py reports the Received request before initialization was complete error as INVALID_PARAMS with empty data). An empty arguments object gets a tool-level validation error with isError: true instead, not -32602.

Why it matters

Once the multiplexer recycles its server instance (credential rotation, idle reap, a crash), every later call on that pooled backend fails until the gateway restarts. The only workaround is to leave Kiro Crew and run the call in a standalone kiro-cli session.

Not a goal

  • Changing the multiplexer. Its replacement instance should get a handshake of its own, and that fix belongs to the multiplexer; this PR makes the gateway recover whatever sits behind the pipe.
  • Recovering servers that signal a lost session some other way. Only the Python SDK's frame is recognised.
  • Retrying initialize, resources/subscribe / unsubscribe, or any method outside the retry set.

What changed (motivation -> approach -> change)

Symptom: -32602 Invalid request parameters with empty data on every call, only under the gateway.

Root cause: the configured command is the multiplexer's thin client. The client keeps its connection to the multiplexer's daemon while the daemon replaces the real server and starts the replacement cold, sending it nothing before the next request. The gateway handshakes each backend once (_handle_initialize) and answers every later stub from _init_result, and kiro-cli sends initialize once per session, so nobody handshakes the new process. A standalone kiro-cli run is a fresh connection with a fresh handshake, which is why it works.

Change, in src/kiro_crew/mcp_gateway/backend.py:

  • The upstream initialize frame is kept on Backend._upstream_init_frame (both send sites: _handle_initialize, prime_initialize).
  • forward_from_stub keeps the frame as it went upstream (identity blocks included) on _PendingRequest.retry_frame, only for methods in _REHANDSHAKE_RETRY_METHODS (tools/call, tools/list, prompts/list, prompts/get, resources/list, resources/read, resources/templates/list, completion/complete). The refusal comes before the SDK dispatches anything, so each is safe to send twice.
  • _route_backend_line: when a response matches _is_lost_session_error and its pending has a retry_frame, _retry_after_rehandshake writes initialize (new gateway id, reply swallowed under _REHANDSHAKE_STUB_SENTINEL), notifications/initialized and the request (new id, same pending fields, retry_frame=None) in ONE write via the new _write_json_lines. One write keeps the retry behind the handshake and keeps a co-tenant's request out of the gap between initialize and initialized, where a Python server refuses everything. The retry is never retried: a second refusal reaches the stub unchanged.
  • Nothing changes when the backend is not ready, has no cached frame, the stub has detached, or the write fails: the first answer is delivered as before.
  • The frame is not proof the session was lost: a request that fails the SDK's ClientRequest validation (non-object arguments, malformed _meta) gets the same frame. Such a request now costs one extra handshake and one retry before its error reaches the caller unchanged. A second initialize on a healthy Python server is harmless: it moves the session to initializing, the initialized in the same write moves it back, and later calls work (see Manual verification). The kiro-cli client builds arguments as an object, so this path should be rare.
  • A WARNING names each recovery, so the next report is one grep away.
  • docs/architecture/mcp.md Troubleshooting gets the symptom and the mechanism.

Backwards compatibility

Compatible: only a response that already fails the call is affected, and only for the one frame the Python SDK sends for an uninitialized session. Every other answer and every other method takes the existing path.

Removes nothing: the diff adds a recovery path and two fields; no key, validator, record or public function is removed or tightened.

Tests

test/test_mcp_gateway_rehandshake.py (new):

  • test_lost_session_is_handshaken_again_and_the_call_retried_once: one write with the original upstream initialize, initialized and the same request under a new id; the refusal never reaches the stub; the initialize reply is swallowed; the retry's answer reaches the stub under its own id; no pending is left.
  • test_a_retry_refused_again_is_delivered_not_retried: no loop.
  • test_other_errors_are_delivered_untouched (3 cases): a tool-filter "Tool X not found" -32602, the same message with non-empty data, and a different code.
  • test_a_method_outside_the_retry_set_is_delivered_untouched
  • test_no_retry_for_a_stub_that_has_detached
  • test_the_retry_carries_the_injected_identity_blocks

The two positive tests fail on origin/main; the negative ones pass on both. test/test_mcp_gateway*.py (49 files) pass locally except test_mcp_gateway_adopted_daemon_repair.py::test_a_stale_survivor_is_repaired_end_to_end, which fails the same way on origin/main in this sandbox (unshare EPERM).

Manual verification

End to end with real processes: a gateway Backend from spawn_backend running a real multiplexer thin client, its real daemon, and a Python FastMCP server (mcp 1.17.0) with one string-argument tool. The server process is killed after the first call; the daemon starts a replacement and the client connection stays up.

origin/main:
first call                     -> ok query='192.0.2.0/24'
after instance replaced        -> {'code': -32602, 'message': 'Invalid request parameters', 'data': ''}
one more                       -> {'code': -32602, 'message': 'Invalid request parameters', 'data': ''}

this branch:
backend pid=4454 server=example-mcp refused tools/call as not initialized; its MCP session was lost behind the pipe (a pooling proxy respawned it cold?) -- re-sending initialize and retrying once
first call                     -> ok query='192.0.2.0/24'
after instance replaced        -> ok query='192.0.2.0/24'
one more                       -> ok query='192.0.2.0/24'

Healthy server, malformed request (arguments a string), then exactly what the gateway sends (initialize + initialized + the same request in one write), then a normal call:

2 {'code': -32602, 'message': 'Invalid request parameters', 'data': ''}   <- malformed, first answer
3 initialize ok                                                          <- re-sent initialize (swallowed by the gateway)
4 {'code': -32602, 'message': 'Invalid request parameters', 'data': ''}   <- retry, delivered to the caller
5 ok query='ok'                                                          <- server still healthy

Raw-frame probe against the same server: arguments set to {}, null or left out gives isError: true (tool validation), never -32602. -32602 with empty data comes back only for a request before initialized, a non-object arguments, or a malformed _meta.

Related Issues

None filed. Reported by a user; root cause reproduced as described above.

Pattern harvest

Not generalizable: the gateway caches one handshake per backend on purpose. The defect is a peer that loses its session behind a live pipe, and recognising that needs the peer's own error frame. No rule can find that statically.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

… session

When the command the gateway spawns is a pooling multiplexer's thin client,
the multiplexer can replace the real server behind the pipe (retired, died)
and start the replacement cold, while the client's own connection stays up.
The gateway sent initialize once for that backend and answers every later
stub from its cache, so nothing handshakes the new process: a Python MCP
server then refuses every request with -32602 "Invalid request parameters"
and empty data until the gateway restarts. A standalone kiro-cli run works
because it is a fresh connection with a fresh handshake.

On exactly that refusal, for a request safe to send twice, the backend now
re-sends its cached initialize, notifications/initialized and the request in
one write, once. The initialize reply is swallowed; the stub sees only the
retry's answer under its own id. Any other error, a second refusal, or a
detached stub is delivered unchanged.
@chenmingwei23
chenmingwei23 requested a review from a team as a code owner October 3, 2026 05:58
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 3, 2026
@github-actions

github-actions Bot commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of e7a0e6f22a85000714a39d7a9fea75acb62a97e7 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

Check before merge: the supplied patch has about 170 files outside backend.py, mcp.md and the new test, so confirm the PR diff against current main is only those three.

What this change ships

Inventory (5 items) — 5 justified

Intent: keep tools from a multiplexer-pooled MCP server working from dashboard chats and subagents after the multiplexer replaces the server process behind them. This is a FIX.

  1. A call that a respawned, never-handshaken server refuses with -32602/empty data now succeeds: the gateway sends initialize again and retries the call once — justified
  2. The same recovery covers the other seven read-only methods in _REHANDSHAKE_RETRY_METHODS (tools/list, prompts/, resources/ reads, completion) — justified
  3. A malformed request to a healthy server now costs one extra handshake and one retry before its unchanged error reaches the caller — justified
  4. The gateway log now carries a WARNING for each recovery — justified
  5. MCP Troubleshooting now documents this symptom and the recovery — justified

[FIRST-PRINCIPLES-REVIEWED] e7a0e6f

Description read

[DESCRIPTION-READ] 49c165e2a7b74c075ae44d3601326f1f4e8d0a9a237d41ea93b251b4235e3ca2

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of e7a0e6f22a85000714a39d7a9fea75acb62a97e7 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Check that no other MCP SDK sends -32602 "Invalid request parameters" with empty data after it has already run a tools/call, since the gateway would then run that call twice.

[DESIGN-REVIEWED] e7a0e6f

Description read

[DESCRIPTION-READ] 83eaca6d7d9b7f06d3421d7f804d9ea6d604c65d5d82c42413170224fd322a8e

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed e7a0e6f22a85000714a39d7a9fea75acb62a97e7 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] e7a0e6f

Verdict parsed from the review's SHA-scoped output markers for commit e7a0e6f22a85000714a39d7a9fea75acb62a97e7.

False positive or not applicable? A repository writer can comment:
/ai-review override fable e7a0e6f22a85000714a39d7a9fea75acb62a97e7: <one-sentence reason>

@github-actions

github-actions Bot commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings (all downgraded on adjudication)

GPT 6.1 flagged blocking issues on e7a0e6f22a85000714a39d7a9fea75acb62a97e7; Opus 5.5 adjudication downgraded every one of them to advisory. Adjudication downgraded all 1 blocking finding(s) to advisory: the remedy each one requires is disproportionate to the harm it prevents. Read them as advice, not as merge conditions.

This comment is updated in place on each push.

Review details

BLOCKING -- website/src/components/markdown/MarkdownTable.tsx:150 -- Widen adds a third action to the table’s button row
<button type="button" data-testid="table-expand" className={markdown-table-expand ${TABLE_ACTION_BTN_CLS}}
Assistant reply containing a top-level table at viewport width ≥880px -> MarkdownTable and its CSS render Widen alongside Copy Markdown and Copy CSV -> three sibling actions occupy one horizontal group.
Anchor: max-two-buttons-per-row
Fix: Keep Widen and one copy-menu trigger in the row; move both copy formats into that menu.

[BLOCK-MERGE-DOWNGRADED] e7a0e6f
[GPT-REVIEWED] e7a0e6f

Adjudication (Opus 5.5) — is blocking on each finding proportionate?

F1: LOW. A third button in a hover-revealed table action row is a visible layout and UX issue, with no data, security or correctness effect.

  • Conditions confirmed: the Widen, Copy Markdown and Copy CSV buttons are siblings at website/src/components/markdown/MarkdownTable.tsx:150, :157 and :161. The same three buttons are already on the base commit 069f4cab4; they landed in feat: opt-in chat table breakout and resizable columns #15547. This PR's diff (git diff --stat 069f4cab4 HEAD) touches only src/kiro_crew/mcp_gateway/backend.py, test/test_mcp_gateway_rehandshake.py and docs/architecture/mcp.md. The max-two-buttons-per-row rule says it "Applies to what the diff ADDS" and grandfathers rows already on base (website/AUTOSDE.yaml:329-331), so its blocking flag doesn't reach this PR.

  • Cost of the real fix: an MCP-gateway PR would have to rework an unrelated frontend component into a dropdown, which breaks the one-change-per-PR rule. That cost is more than a LOW, already-merged UI issue justifies. Recovery path: none needed here; the follow-up belongs to the feat: opt-in chat table breakout and resizable columns #15547 owner.

    [ADJUDICATION] e7a0e6f total=1 uphold=0 downgrade=1
    DOWNGRADE F1 website/src/components/markdown/MarkdownTable.tsx:150 reason=disproportionate-remedy
    [GPT-ADJUDICATED] e7a0e6f

False positive or not applicable? A repository writer can comment:
/ai-review override gpt e7a0e6f22a85000714a39d7a9fea75acb62a97e7: <one-sentence reason>

@github-actions

github-actions Bot commented Oct 3, 2026

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ✅ PASS

UX-level review of e7a0e6f22a85000714a39d7a9fea75acb62a97e7 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

UX-Verdict: PASS

Nothing to check.

[UX-REVIEWED] e7a0e6f

Description read

[DESCRIPTION-READ] dbf1c77aa450abdb775a305a11e994ccaefac56bba7eaa3ef9818666d972c541

That sha256 names everything this verdict read: the PR description, plus the 3 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 3, 2026
@iamwhatever
iamwhatever merged commit e698e8c into main Oct 3, 2026
192 of 200 checks passed
@iamwhatever
iamwhatever deleted the fix/mcp-gateway-rehandshake-lost-session branch October 3, 2026 07:57
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants