Skip to content

fix(cron): reuse one MCP server per name across a script cron run - #16661

Merged
iamwhatever merged 1 commit into
kirodotdev:mainfrom
unstablebrainiac:fix/cron-script-reuse-mcp-servers
Oct 4, 2026
Merged

iamwhatever merged 1 commit into
kirodotdev:mainfrom
unstablebrainiac:fix/cron-script-reuse-mcp-servers

Conversation

@unstablebrainiac

@unstablebrainiac unstablebrainiac commented Oct 4, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A script cron's calls to the same MCP server within one run share one server process, so a server that signs in when it starts signs in once per run instead of once per call.

ctx.call_tool started a new MCP server for every call and stopped it after the answer. A Slack MCP server signs in to Slack each time it starts, and Slack refuses sign-ins that come close together with HTTP 429. A script cron that ran every minute and made several Slack calls per run had 71 of its 97 Slack calls refused in 25 minutes. The first call of each run nearly always worked; the calls right after it failed.

Why it matters

Any script cron that calls a sign-in server more than once per run hits this, and the refusals grow with the number of calls. A refused call is lost work: in the case above, the cron missed most of the Slack messages it existed to watch during a live incident. Every call also paid a full server start and initialize handshake.

Not a goal

  • Sharing a server across runs or across jobs. Each run still starts its own.
  • Changing how chat sessions or gatewayd start and pool MCP servers.
  • Retrying refused calls, or any rate limiting.
  • Changing McpToolClient's JSON-RPC handling.

What changed (motivation → approach → change)

A script run now keeps the MCP server it started and reuses it for the run's later calls to that server. A server that signs in when it starts then signs in once per run, not once per call, which is what got most of a Slack cron's calls refused. kirocrew cron preview runs a script the same way.

When Before After
First call to a server start, call, stop start, call, keep
Later call to that server start a new one use the kept one
A call fails its server stops its server stops; the next call starts a fresh one
The kept server has exited n/a the next call replaces it
Two calls to one server at once two servers two servers; one is kept afterwards
The script function returns nothing running the kept servers are stopped

The cause was ScriptContext.call_tool in src/kiro_crew/cron_script.py building an McpToolClient per call and closing it in finally; the cron preview's context in src/kiro_crew/cli_commands.py did the same. Both now hand their calls to KeptMcpServers, a small class in cron_script.py. It keeps at most one client per server name, and only after a call to it succeeds, so a failed call never hands a broken server to the next one. A call made while another call to the same server is still running gets a server of its own, so a call a script has given up on never blocks a later one. McpToolClient.is_running() lets a kept client whose process exited be replaced. The launcher calls ctx.close() in a finally, whatever the script raised, and the preview closes its context the same way; close() never raises. The cost is that a kept server stays up until the run ends, not for one call. The cron spec, docs/system-specs/modules/learn-cron-dashboard.md, has a new bullet on this lifetime.

Backwards compatibility

Compatible: a script gets the same results from the same calls. Only when servers start and stop changes.

Removes nothing: no key, function or record is removed or renamed. cron_script.py gains KeptMcpServers, ScriptContext gains close() and McpToolClient gains is_running().

Tests

  • TestScriptContextKeepsServers in test/test_cron_script.py: three calls share one server; each server name gets its own; a failed call stops its server and the next starts another; an exited server is replaced; a call made during another gets its own server and only one is kept; nothing is kept after close(); close() stops every server even when one fails to stop.
  • test_the_launcher_closes_the_context_however_the_script_ends: a real script child, started in its own temporary directory, returns, raises Skip or raises an error, and the launcher closes the context each time.
  • test_call_tool_keeps_one_server_per_name_until_the_script_returns in test/test_cron_preview_cmd.py: two previewed calls to one server start one client, closed once after the script returned.
  • Two existing success-path tests (test/test_cron_script.py, test/test_cron_script_more_coverage.py) now check that the client stays open until close().

Each mechanism reverted on its own turns a named test red: no reuse, reusing an exited server, replacing a kept server, keeping after close(), a failing stop aborting close(), the launcher not closing, keeping a server after a failed call, and the preview starting a server per call.

Manual verification

Ran the change against a real stdio MCP server process that reports its pid: three ctx.call_tool calls were answered by one pid, ctx.close() stopped that process, and a call after close() started a new process and stopped it again.

Related Issues

Closes #16659

Pattern harvest

Not generalizable: the two script contexts that started a server per call, the cron run's and the preview's, now share KeptMcpServers, which is the only remaining place that builds an McpToolClient. Chat sessions keep their servers for the session, and gatewayd pools them.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@unstablebrainiac
unstablebrainiac requested a review from a team as a code owner October 4, 2026 05:18
@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: action required A blocking check or review needs attention labels Oct 4, 2026
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 4, 2026
@github-actions

github-actions Bot commented Oct 4, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5, fork) — 🟡 CONCERNS

Design-level review of a035cbdc9f5a3e708468b5460ab31060edc77a86 via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: CONCERNS

A tool-level error closes the kept server, so a cron whose calls return isError still signs in again for every call.

Watch

  • Ordinary tool errors restart the server. KeptMcpServers.call_tool keeps a client only when client.call_tool returns. But McpToolClient.call_tool raises RuntimeError("MCP tool error: …") for a JSON-RPC error reply and for an isError result, and both come from a server that is still healthy. Each one closes the server, and the next call starts a fresh one and signs in again. The case this PR targets is a Slack cron hitting rate limits. If Slack's refusals reach the script as isError results (missing channel, rate limited), every failure brings another sign-in, which draws more 429s: the per-call restart the PR set out to remove comes back. The stated reason, "a failed call never hands a broken server to the next one", only covers transport failures: a disconnect, the 1000-line cap, or an exited process (is_running() already catches that).
    Clears when: a tool-level error (a JSON-RPC error reply or an isError result from a server that is still running) keeps the server, and only transport or protocol failures close it, with a test that pins it.

[DESIGN-REVIEWED] a035cbd

Description read

[DESCRIPTION-READ] 75303d26b345f282eebd97c96663f9f10924538fc09a54ea716d9e7d76c7f9f8

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 4, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5, fork) — ✅ PASS

Premise-level review of a035cbdc9f5a3e708468b5460ab31060edc77a86 via the fork AI-review pipeline — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

Before merge, check that a run killed at the job timeout still stops its kept servers. ctx.close() sits in a finally that a killed child never runs.

What this change ships

Inventory (7 items) — 7 justified

Intent: let a script cron call a sign-in MCP server like Slack several times per run without most calls refused with HTTP 429 (#16659). FIX.

  1. Within one run, a script cron's calls to the same server reuse one server process instead of starting one per call — justified
  2. kirocrew cron preview reuses servers the same way, so previewing the issue's script no longer draws the same 429s — justified
  3. A failed call stops its server and the next call starts a fresh one — justified
  4. A kept server whose process has exited is replaced on the next call (McpToolClient.is_running(), one consumer) — justified
  5. A call made while another call to the same server is running gets its own server, and only one is kept afterwards — justified
  6. The launcher stops all kept servers once the script function returns, raises or skips (ScriptContext.close() in a finally) — justified
  7. The cron spec gains a bullet on how long a kept server lives — justified

[FIRST-PRINCIPLES-REVIEWED] a035cbd

Description read

[DESCRIPTION-READ] 9bc551facf7391df40d52238ed5c48a18845251ac35836e0baa0ef56a2ca5168

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 4, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review (fork) — ✅ no blocking findings

Reviewed a035cbdc9f5a3e708468b5460ab31060edc77a86 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] a035cbd

@github-actions

github-actions Bot commented Oct 4, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review (fork) — ✅ no blocking findings

Reviewed a035cbdc9f5a3e708468b5460ab31060edc77a86 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] a035cbd

@unstablebrainiac
unstablebrainiac force-pushed the fix/cron-script-reuse-mcp-servers branch from bea332e to 0b18b91 Compare October 4, 2026 06:33
@unstablebrainiac

Copy link
Copy Markdown
Contributor Author

mechanism: new KeptMcpServers class in cron_script.py, shared by ScriptContext and the cron preview's context

  • span=6dc55df8dc3b: kirocrew cron preview still started a server per call

Outcome: fixed in 0b18b91.
The keep-and-reuse logic moved out of ScriptContext into KeptMcpServers. ScriptContext.call_tool and the preview's _LiveTestCtx.call_tool both call it, and both contexts' close() stops the kept servers once the script returns, so the preview's comment claiming the production lifecycle is true again.
test_call_tool_keeps_one_server_per_name_until_the_script_returns pins it: two previewed calls start one client, closed once after the script returned.
The PR body's Pattern harvest now names both contexts.

@unstablebrainiac

Copy link
Copy Markdown
Contributor Author

self-added: yes
mechanism: the launcher test starts its script child in tmp_path and asserts the child's working directory

  • span=4d0499b2470e: the launcher test's script child ran in the checkout

Outcome: fixed in 0b18b91.
test_the_launcher_closes_the_context_however_the_script_ends now wraps cron_script.popen_limited with cwd=tmp_path, so the real script child starts in the test's own temporary directory, not the checkout.
The script's replacement close writes Path.cwd() to its marker file and the test asserts it equals tmp_path, so a child started anywhere else fails all three cases.

@github-actions github-actions Bot added the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Oct 4, 2026
@unstablebrainiac
unstablebrainiac force-pushed the fix/cron-script-reuse-mcp-servers branch from 0b18b91 to da70999 Compare October 4, 2026 07:59
@unstablebrainiac

Copy link
Copy Markdown
Contributor Author

Rebased onto af2619f. #16280 conflicted with this PR in docs/system-specs/modules/learn-cron-dashboard.md only, where both added a bullet at the same spot; both bullets are kept and the code and test changes are unchanged. The three backend reds on the previous head were main's own tests, fixed there by #16695 and #16717; CI re-running.

@github-actions github-actions Bot removed the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Oct 4, 2026
@unstablebrainiac
unstablebrainiac force-pushed the fix/cron-script-reuse-mcp-servers branch from da70999 to d79106b Compare October 4, 2026 08:40
@unstablebrainiac

Copy link
Copy Markdown
Contributor Author

Rebased onto 3f960a1 with the diff unchanged, to re-run CI. On da70999, Backend Lint & Type Check (3.12) hit its 25-minute job timeout during mypy: isort and flake8 had passed, but the runner spent 14.5 minutes in the black step. A fork author cannot re-run a single job. Main had also moved in files that import this PR's modules (session_ledger handlers, tools and tests).

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 4, 2026
ctx.call_tool started a new MCP server for every call and stopped it
afterwards. A server that signs in to a service when it starts signed in
once per call, and a Slack server doing that from a once-a-minute cron
had most of its calls refused with HTTP 429.

KeptMcpServers keeps a server whose call succeeds for the run's next
call to the same server name. A failed call stops its server, a kept
server whose process exited is replaced, and a call made while another
call to the same server is running gets a server of its own.
ScriptContext and the kirocrew cron preview context both use it, and
both stop their kept servers once the script function returns.
@unstablebrainiac
unstablebrainiac force-pushed the fix/cron-script-reuse-mcp-servers branch from d79106b to a035cbd Compare October 4, 2026 09:25
@unstablebrainiac

Copy link
Copy Markdown
Contributor Author

Rebased onto 9757a35 with the diff unchanged, to re-run CI. The only red on d79106b was Backend Tests (Windows) (5), failing one test: test/test_session_start_gate.py::test_a_gate_queued_claim_already_counts_as_initializing ("queued claim never opened its init scope"). It is an ACP runtime asyncio timing test that imports nothing this PR changes, and it passes on repeated local runs. Every review lane had passed on that head.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention labels Oct 4, 2026
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 4, 2026
@unstablebrainiac

Copy link
Copy Markdown
Contributor Author

self-added: yes
mechanism: proposed McpToolError, a RuntimeError subclass, so KeptMcpServers can keep a server that answered with a tool error

  • span=37ac5bc9148f: ordinary tool errors restart the server

Outcome: agreed, and deferred to a follow-up rather than pushed onto this green head.
This head is still better than main, where every call started and stopped its own server: a tool error here costs one extra sign-in, which is what every call costs on main.
The fix is ready and tested: McpToolClient.call_tool raises McpToolError, with the same message, for an error reply or an isError result, and KeptMcpServers keeps the server after one, while a call with no answer still stops it. I can push it onto this PR on request, or open it as a follow-up once this merges.

@iamwhatever iamwhatever left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed: a script cron run now keeps one MCP server per name for the run (KeptMcpServers) instead of spawning one per call, so a server that signs in on start signs in once per run (fixes the Slack 429s in #16659). A failed call stops its server, an exited one is replaced, a concurrent call to the same name gets its own server, and the launcher calls ctx.close() in a finally however the script ends, which stops every kept server; preview uses the same lifecycle. No change to which processes are signalled beyond the ones the run itself started. Advisory: if the sandboxed script process is killed by its timeout, the finally does not run and kept servers live until their stdin closes; worth one sentence in the spec.

@iamwhatever
iamwhatever enabled auto-merge (squash) October 4, 2026 16:45
@iamwhatever
iamwhatever merged commit 0d4385d into kirodotdev:main Oct 4, 2026
101 checks passed
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 4, 2026
@unstablebrainiac

Copy link
Copy Markdown
Contributor Author

Follow-up in #16905: a call the server answers with a tool error now keeps the server, which is the Design point deferred above, and the cron spec gains the timeout sentence.

One correction to the timeout note: a timeout kill does stop kept servers. Each runs in the script child's process group, which _kill_proc_group SIGKILLs on POSIX, or in its process tree, which taskkill /T reaps on Windows. Only a server that starts a session of its own outlives the kill, and it lives until it reads end-of-file on its stdin. A timed-out run on Linux showed both cases.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

cron: a script cron starts a new MCP server for every ctx.call_tool

2 participants