Skip to content

fix(mcp-gateway): give a server reaped at the initialize deadline a longer one next time - #39546

Merged
buluoray merged 1 commit into
mainfrom
crew/slack/fix-core-init-fast
Nov 19, 2026
Merged

buluoray merged 1 commit into
mainfrom
crew/slack/fix-core-init-fast

Conversation

@buluoray

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A Crew MCP server whose cold start runs a little past the 30 s initialize deadline starts on a later attempt, instead of failing every chat.

On some hosts every KAS chat fails with:

KAS managed MCP initialization failed: kirocrew-core: failed (backend init failed: initialize did not complete within 30s)

Updating and restarting do not help. A Windows install measured a cold kirocrew mcp-core start at 33.9 s, just over the deadline (#30915).

Why it matters

When it happens, no chat works on that machine. The gateway kills the slow backend at 30 s, and the next attempt starts a new cold process with the same 30 s. So every attempt is cut at the same point, and the server never comes up.

Not a goal

What changed (motivation → approach → change)

A backend reaped at the deadline is killed, so its respawn is cold again. The fix makes the next spawn for the same pool key wait longer.

Spawn for one pool key Deadline
First, or after a handshake that completed configured (30 s floor), as today
After 1 reap at the deadline 60 s
After 2 or more reaps in a row 120 s (cap)

Admission.initialize_timeout_for picks the deadline. Admission.track_initialize watches each spawned backend's first handshake: a deadline reap adds one, a completed handshake clears the count. gatewayd._acquire_backend uses both. A configured value above the cap is never lowered. A backend that never answers still fails, only later.

Backwards compatibility

Compatible: the first spawn of every key keeps today's deadline; only a respawn after a deadline reap waits longer.

Removes nothing: the change only lengthens a deadline for a key whose last spawns were reaped.

Tests

test/test_mcp_gateway_spawn_gate.py::TestAcquireBackendOrder::test_a_key_reaped_at_the_initialize_deadline_gets_a_longer_one_next drives the real _acquire_backend and the backend's real deadline timer: three reaps in a row give 10 (configured), 60, 120; a fourth spawn stays at the cap; a completed handshake resets the key to the configured value.

Proof: not provable -- other: prove.py reverts each admission.py hunk alone, which removes an import or a name and fails collection; reverting only the gatewayd.py line that reads initialize_timeout_for turns the test red with assert [10.0, 10.0, 10.0] == [10.0, 60.0, 120.0].

Determinism: the new test passed 20 of 20 runs alone and in the full file with -n0 and -n4. test/test_flake_pattern_ratchet.py passes.

Manual verification

N/A — the real-machine run below is the manual check.

Real-machine verification

A real slow MCP server (sleeps 40 s before it reads initialize) spawned through the real gatewayd._acquire_backend and spawn_backend, with the shipped 10 s config lifted to the 30 s floor, requested up to three times. Script: a 60-line harness, posted in a PR comment.

Machine: own computer, Amazon Linux 2023 (kernel 6.12), x86_64, 32 vCPU / 247 GB
Before (main 906ee87): PYTHONPATH=src python realmachine.py -> attempt 1: deadline=30s state=failed, attempt 2: deadline=30s state=failed, attempt 3: deadline=30s state=failed
After (head 36a01e9): PYTHONPATH=src python realmachine.py -> attempt 1: deadline=30s state=failed after=30.1s, attempt 2: deadline=60s state=ready after=40.0s
Cleanup: own computer; no slow_mcp.py process left (pgrep -af slow_mcp.py empty), scratch data home only.

Related Issues

Closes #30915

Sibling sweep

Swept: git grep -n "initialize_timeout_secs" -- src/kiro_crew/mcp_gateway/
Sibling: src/kiro_crew/mcp_gateway/gatewayd.py:1286 -- not affected -- the initialize watcher reads backend.initialize_timeout_secs, which is the stretched value
Sibling: src/kiro_crew/mcp_gateway/stub_link.py:202 -- not affected -- the replayed initialize on a reconnect is answered from the gateway's cache or the same backend handshake

Pattern harvest

Not generalizable: the defect is one retry path that reused the exact deadline that just failed; no other retry in the gateway kills and respawns at a fixed deadline.

Checklist

  • A Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...); PR Hygiene does not count commits
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@buluoray
buluoray requested review from a team and KiroCrewCaptain as code owners November 19, 2026 17:50
@buluoray
buluoray requested a review from iamwhatever November 19, 2026 17:50
@buluoray

Copy link
Copy Markdown
Contributor Author

Real-machine harness used for the ## Real-machine verification section. slow_mcp.py sleeps for its argument (40 s) before it reads initialize; realmachine.py drives it through the real gatewayd._acquire_backend and spawn_backend.

slow_mcp.py:

"""A stdio MCP server whose cold start takes SECS seconds before it reads initialize."""
import json
import sys
import time

time.sleep(float(sys.argv[1]))
for line in sys.stdin:
    msg = json.loads(line)
    if msg.get("method") == "initialize":
        result = {"protocolVersion": "2025-06-18", "capabilities": {}, "serverInfo": {"name": "slow", "version": "0"}}
        sys.stdout.write(json.dumps({"jsonrpc": "2.0", "id": msg["id"], "result": result}) + "\n")
        sys.stdout.flush()

realmachine.py:

"""Real processes through the real gateway spawn path: a server whose cold start
takes COLD seconds (over the 30 s initialize floor) is requested ATTEMPTS times.

Prints, per attempt, the deadline the backend got and whether its handshake
completed. Run from a checkout: PYTHONPATH=src python realmachine.py
"""
import asyncio
import os
import sys
import time
import uuid

from kiro_crew.mcp_gateway import admission as adm
from kiro_crew.mcp_gateway import gatewayd as gw
from kiro_crew.mcp_gateway import host_budget as hb
from kiro_crew.mcp_gateway.pool import BackendPool, PoolKey

HERE = os.path.dirname(os.path.abspath(__file__))
COLD = float(os.environ.get("COLD", "40"))
ATTEMPTS = int(os.environ.get("ATTEMPTS", "3"))


def resolver(key):
    return sys.executable, [os.path.join(HERE, "slow_mcp.py"), str(COLD)], dict(os.environ), HERE


async def main() -> None:
    pool = BackendPool(max_backends=4)
    admission = adm.Admission(
        gate=adm.SpawnGate(2),
        budget=hb.HostBudget(hb.HostBudgetLimits(max_procs=0)),
        initialize_timeout_secs=10.0,  # the shipped default; the floor lifts it to 30
        spawn_queue_wait_secs=30.0,
    )
    key = PoolKey(
        server_name="slow-mcp",
        command_args_hash="a" * 8,
        effective_env_hash="e" * 8,
        work_dir=HERE,
        binary_version="1",
        os_uid=os.getuid(),
    )
    for attempt in range(1, ATTEMPTS + 1):
        t0 = time.monotonic()
        backend, _ = await gw._acquire_backend(pool, key, resolver, admission=admission)
        stub = str(uuid.uuid4())
        await backend.attach_stub(stub)
        await backend._handle_initialize(
            stub, {"jsonrpc": "2.0", "id": 1, "method": "initialize", "params": {}}
        )
        await backend._init_done_event.wait()
        print(
            f"attempt {attempt}: deadline={backend.initialize_timeout_secs:g}s "
            f"state={backend._init_state} after={time.monotonic() - t0:.1f}s",
            flush=True,
        )
        if backend._init_state == "ready":
            await backend.shutdown()
            break
        await backend.process.wait()
    await admission.close()


asyncio.run(main())

@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 19, 2026
@github-actions

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 36a01e98da5035415071fac9b33bacb072ccf6d4 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 36a01e98da5035415071fac9b33bacb072ccf6d4 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: the change fixes the actual cause, which is that every respawn used the same deadline that had just failed, and the real-machine run shows the second attempt starting.

Notes

  • The reap count lives only in the gateway daemon's memory. Each time the daemon restarts, the first attempt goes back to 30 s, so a host whose cold start always runs past 30 s fails the first chat after every restart.
  • Deadline reaps are only counted when admission, charge and permit are all present. Spawns on any other path keep the fixed deadline and are never stretched.

[DESIGN-REVIEWED] 36a01e9

Description read

[DESCRIPTION-READ] 2890b49e592c4e4ccc747a4cdd542338f26151d0d486425d1b4f957484887f54

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of 36a01e98da5035415071fac9b33bacb072ccf6d4 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

The fix targets the actual cause: the respawn reused the same deadline that had just failed. It has a linked issue (#30915), a test that fails on base, and a before/after run on a real machine.

Notes

  • Item 3 — duplicate of admission.gate.watch_initialize (gatewayd.py:1281): track_initialize starts a second task per backend that awaits the same _init_done_event / process.wait and reads the same _init_state. The existing watcher is skipped on prewarm, and that is the only gap. Subtraction: move the reap-count update into the existing first-handshake watcher (run it on prewarm too, with no permit to settle) and delete track_initialize.

What this change ships

Inventory (5 items) — 4 justified

Intent: let an MCP server whose cold start runs slightly past 30 s come up on a later attempt, instead of every chat failing. FIX.

  1. After one deadline reap, the next spawn for that server waits 60 s; after two or more in a row, 120 s (cap) — justified
  2. A handshake that completes resets that server's deadline to the configured value — justified
  3. A new background watcher per spawned backend counts deadline reaps — duplicate of gate.watch_initialize signals
  4. New 120 s cap constant INIT_DEADLINE_STRETCH_CAP_SECS — justified
  5. The in-memory reap count stops at 8 — justified

[FIRST-PRINCIPLES-REVIEWED] 36a01e9

Description read

[DESCRIPTION-READ] 04658be04562816e5f2912f8d7fe449156a8c69acfaab3abecd9480681f800d9

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 36a01e98da5035415071fac9b33bacb072ccf6d4 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 36a01e9

Verdict parsed from the review's SHA-scoped output markers for commit 36a01e98da5035415071fac9b33bacb072ccf6d4.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 36a01e98da5035415071fac9b33bacb072ccf6d4 false-positive: <reason, 20 to 500 characters>

@github-actions

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 36a01e98da5035415071fac9b33bacb072ccf6d4 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 36a01e9

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 36a01e98da5035415071fac9b33bacb072ccf6d4 false-positive: <reason, 20 to 500 characters>

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Nov 19, 2026
@buluoray
buluoray enabled auto-merge (squash) November 19, 2026 19:38
@KiroCrewCaptain

Copy link
Copy Markdown
Collaborator

Code change approver recommendation: A person should decide

Why: the decider thinks the change is probably fine but estimates a 15% chance that this change introduces a defect, above the 12% limit for approving on its own.

This fixes respawns reusing the same 30 s initialize deadline that had just failed. A key reaped at that deadline now gets 60 s on the next spawn, then 120 s (the cap). A completed handshake resets it, and a key with no reaps keeps the configured value, so first spawns behave as on main. Only a failure whose reason starts with INITIALIZE_DEADLINE_REASON adds a reap, and the deadline timer marks the backend failed before it kills the process, so the watcher sees the result before the exit. The gate watcher and replacement priming both read the stretched value from Backend.initialize_timeout_secs. KAS's single retry after a start timeout gets 60 s, which covers the 33.9 s cold start in the issue. The test drives the real _acquire_backend and the real deadline timer, and fails without the gatewayd.py change. CI and all review lanes passed. The missing spec update is noted but not blocking.

Tier: 2, held for a person: the automated review did not clear it.

An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit 36a01e98da · 146 changed lines. If this recommendation is wrong, reply here: replies are how the approver is evaluated.

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The review checked this head against the code at each call site. A first attempt keeps today's deadline. After one deadline reap the next attempt gets 60 s, then 120 s, and a completed handshake resets the key. An operator value above the cap is never lowered. Only the deadline-reap reason increments the count: a crash in the handshake, a respawn timeout and an early stub exit do not. The tracker task joins the reaper set, so Admission.close cancels it. The new test passed 8 of 8 runs alone and once with the full file. Reverting the wiring, the tracker call, the reset or the cap each turns it red. The test also drives the real _acquire_backend and the backend's real _init_deadline timer, with no mocked deadline math.

Optional:

  • The owning specs still describe the initialize deadline as the configured value floored to 30 s (docs/architecture/mcp.md near line 1789, docs/system-specs/modules/config.md near line 3619, docs/system-specs/modules/acp-client.md near line 3546). One sentence in each about the per-key stretch, the 120 s cap and the reset would keep them in step with the code.
  • The "any other reason does not stretch the deadline" block checks a key that never spawned. With the reason check loosened to elif state == "failed": the file still passes. A real crash on the tracked key would pin that rule.
  • A stretched deadline that succeeds leaves no trace in the logs or the admission snapshot. An info line when a key's count goes up would save time when someone debugs a slow server.

@buluoray
buluoray merged commit 8548bc5 into main Nov 19, 2026
158 of 160 checks passed
@buluoray
buluoray deleted the crew/slack/fix-core-init-fast branch November 19, 2026 20:04
buluoray added a commit that referenced this pull request Nov 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

readiness: passed Eligible automated validation passed for the current revision

Projects

None yet

Development

Successfully merging this pull request may close these issues.

KAS: kirocrew-core never completes initialize within the 30s floor, so every chat fails and a restart does not help

2 participants