Repository navigation
fix(mcp-gateway): give a server reaped at the initialize deadline a longer one next time - #39546
Conversation
…onger one next time
|
Real-machine harness used for the
"""A stdio MCP server whose cold start takes SECS seconds before it reads initialize."""
import json
import sys
import time
time.sleep(float(sys.argv[1]))
for line in sys.stdin:
msg = json.loads(line)
if msg.get("method") == "initialize":
result = {"protocolVersion": "2025-06-18", "capabilities": {}, "serverInfo": {"name": "slow", "version": "0"}}
sys.stdout.write(json.dumps({"jsonrpc": "2.0", "id": msg["id"], "result": result}) + "\n")
sys.stdout.flush()
"""Real processes through the real gateway spawn path: a server whose cold start
takes COLD seconds (over the 30 s initialize floor) is requested ATTEMPTS times.
Prints, per attempt, the deadline the backend got and whether its handshake
completed. Run from a checkout: PYTHONPATH=src python realmachine.py
"""
import asyncio
import os
import sys
import time
import uuid
from kiro_crew.mcp_gateway import admission as adm
from kiro_crew.mcp_gateway import gatewayd as gw
from kiro_crew.mcp_gateway import host_budget as hb
from kiro_crew.mcp_gateway.pool import BackendPool, PoolKey
HERE = os.path.dirname(os.path.abspath(__file__))
COLD = float(os.environ.get("COLD", "40"))
ATTEMPTS = int(os.environ.get("ATTEMPTS", "3"))
def resolver(key):
return sys.executable, [os.path.join(HERE, "slow_mcp.py"), str(COLD)], dict(os.environ), HERE
async def main() -> None:
pool = BackendPool(max_backends=4)
admission = adm.Admission(
gate=adm.SpawnGate(2),
budget=hb.HostBudget(hb.HostBudgetLimits(max_procs=0)),
initialize_timeout_secs=10.0, # the shipped default; the floor lifts it to 30
spawn_queue_wait_secs=30.0,
)
key = PoolKey(
server_name="slow-mcp",
command_args_hash="a" * 8,
effective_env_hash="e" * 8,
work_dir=HERE,
binary_version="1",
os_uid=os.getuid(),
)
for attempt in range(1, ATTEMPTS + 1):
t0 = time.monotonic()
backend, _ = await gw._acquire_backend(pool, key, resolver, admission=admission)
stub = str(uuid.uuid4())
await backend.attach_stub(stub)
await backend._handle_initialize(
stub, {"jsonrpc": "2.0", "id": 1, "method": "initialize", "params": {}}
)
await backend._init_done_event.wait()
print(
f"attempt {attempt}: deadline={backend.initialize_timeout_secs:g}s "
f"state={backend._init_state} after={time.monotonic() - t0:.1f}s",
flush=True,
)
if backend._init_state == "ready":
await backend.shutdown()
break
await backend.process.wait()
await admission.close()
asyncio.run(main()) |
UX Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing blocks: the change fixes the actual cause, which is that every respawn used the same deadline that had just failed, and the real-machine run shows the second attempt starting. Notes
[DESIGN-REVIEWED] 36a01e9 Description read[DESCRIPTION-READ] 2890b49e592c4e4ccc747a4cdd542338f26151d0d486425d1b4f957484887f54 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS The fix targets the actual cause: the respawn reused the same deadline that had just failed. It has a linked issue (#30915), a test that fails on base, and a before/after run on a real machine. Notes
What this change shipsInventory (5 items) — 4 justifiedIntent: let an MCP server whose cold start runs slightly past 30 s come up on a later attempt, instead of every chat failing. FIX.
[FIRST-PRINCIPLES-REVIEWED] 36a01e9 Description read[DESCRIPTION-READ] 04658be04562816e5f2912f8d7fe449156a8c69acfaab3abecd9480681f800d9 That sha256 names the PR description this verdict read. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
Code change approver recommendation: A person should decideWhy: the decider thinks the change is probably fine but estimates a 15% chance that this change introduces a defect, above the 12% limit for approving on its own. This fixes respawns reusing the same 30 s initialize deadline that had just failed. A key reaped at that deadline now gets 60 s on the next spawn, then 120 s (the cap). A completed handshake resets it, and a key with no reaps keeps the configured value, so first spawns behave as on main. Only a failure whose reason starts with INITIALIZE_DEADLINE_REASON adds a reap, and the deadline timer marks the backend failed before it kills the process, so the watcher sees the result before the exit. The gate watcher and replacement priming both read the stretched value from Backend.initialize_timeout_secs. KAS's single retry after a start timeout gets 60 s, which covers the 33.9 s cold start in the issue. The test drives the real _acquire_backend and the real deadline timer, and fails without the gatewayd.py change. CI and all review lanes passed. The missing spec update is noted but not blocking. Tier: 2, held for a person: the automated review did not clear it. An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit |
KiroCrewCaptain
left a comment
There was a problem hiding this comment.
The review checked this head against the code at each call site. A first attempt keeps today's deadline. After one deadline reap the next attempt gets 60 s, then 120 s, and a completed handshake resets the key. An operator value above the cap is never lowered. Only the deadline-reap reason increments the count: a crash in the handshake, a respawn timeout and an early stub exit do not. The tracker task joins the reaper set, so Admission.close cancels it. The new test passed 8 of 8 runs alone and once with the full file. Reverting the wiring, the tracker call, the reset or the cap each turns it red. The test also drives the real _acquire_backend and the backend's real _init_deadline timer, with no mocked deadline math.
Optional:
- The owning specs still describe the initialize deadline as the configured value floored to 30 s (docs/architecture/mcp.md near line 1789, docs/system-specs/modules/config.md near line 3619, docs/system-specs/modules/acp-client.md near line 3546). One sentence in each about the per-key stretch, the 120 s cap and the reset would keep them in step with the code.
- The "any other reason does not stretch the deadline" block checks a key that never spawned. With the reason check loosened to
elif state == "failed":the file still passes. A real crash on the tracked key would pin that rule. - A stretched deadline that succeeds leaves no trace in the logs or the admission snapshot. An info line when a key's count goes up would save time when someone debugs a slow server.
Problem / Motivation
Goal: A Crew MCP server whose cold start runs a little past the 30 s initialize deadline starts on a later attempt, instead of failing every chat.
On some hosts every KAS chat fails with:
Updating and restarting do not help. A Windows install measured a cold
kirocrew mcp-corestart at 33.9 s, just over the deadline (#30915).Why it matters
When it happens, no chat works on that machine. The gateway kills the slow backend at 30 s, and the next attempt starts a new cold process with the same 30 s. So every attempt is cut at the same point, and the server never comes up.
Not a goal
kirocrew-corealready answersinitializebefore it loads its tool handlers (perf: list MCP tools before loading tool handlers #35293, perf: relay own MCP servers' tool lists from the gateway #35833).What changed (motivation → approach → change)
A backend reaped at the deadline is killed, so its respawn is cold again. The fix makes the next spawn for the same pool key wait longer.
Admission.initialize_timeout_forpicks the deadline.Admission.track_initializewatches each spawned backend's first handshake: a deadline reap adds one, a completed handshake clears the count.gatewayd._acquire_backenduses both. A configured value above the cap is never lowered. A backend that never answers still fails, only later.Backwards compatibility
Compatible: the first spawn of every key keeps today's deadline; only a respawn after a deadline reap waits longer.
Removes nothing: the change only lengthens a deadline for a key whose last spawns were reaped.
Tests
test/test_mcp_gateway_spawn_gate.py::TestAcquireBackendOrder::test_a_key_reaped_at_the_initialize_deadline_gets_a_longer_one_nextdrives the real_acquire_backendand the backend's real deadline timer: three reaps in a row give 10 (configured), 60, 120; a fourth spawn stays at the cap; a completed handshake resets the key to the configured value.Proof: not provable -- other: prove.py reverts each admission.py hunk alone, which removes an import or a name and fails collection; reverting only the gatewayd.py line that reads
initialize_timeout_forturns the test red withassert [10.0, 10.0, 10.0] == [10.0, 60.0, 120.0].Determinism: the new test passed 20 of 20 runs alone and in the full file with
-n0and-n4.test/test_flake_pattern_ratchet.pypasses.Manual verification
N/A — the real-machine run below is the manual check.
Real-machine verification
A real slow MCP server (sleeps 40 s before it reads
initialize) spawned through the realgatewayd._acquire_backendandspawn_backend, with the shipped 10 s config lifted to the 30 s floor, requested up to three times. Script: a 60-line harness, posted in a PR comment.Machine: own computer, Amazon Linux 2023 (kernel 6.12), x86_64, 32 vCPU / 247 GB
Before (main 906ee87):
PYTHONPATH=src python realmachine.py->attempt 1: deadline=30s state=failed,attempt 2: deadline=30s state=failed,attempt 3: deadline=30s state=failedAfter (head 36a01e9):
PYTHONPATH=src python realmachine.py->attempt 1: deadline=30s state=failed after=30.1s,attempt 2: deadline=60s state=ready after=40.0sCleanup: own computer; no
slow_mcp.pyprocess left (pgrep -af slow_mcp.pyempty), scratch data home only.Related Issues
Closes #30915
Sibling sweep
Swept:
git grep -n "initialize_timeout_secs" -- src/kiro_crew/mcp_gateway/Sibling: src/kiro_crew/mcp_gateway/gatewayd.py:1286 -- not affected -- the initialize watcher reads backend.initialize_timeout_secs, which is the stretched value
Sibling: src/kiro_crew/mcp_gateway/stub_link.py:202 -- not affected -- the replayed initialize on a reconnect is answered from the gateway's cache or the same backend handshake
Pattern harvest
Not generalizable: the defect is one retry path that reused the exact deadline that just failed; no other retry in the gateway kills and respawns at a fixed deadline.
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...); PR Hygiene does not count commits