Skip to content

fix(mcp): park waiting tools off the worker so a wait stalls nobody (0.9.0 backport of #18841) - #20631

Merged
chenmingwei23 merged 1 commit into
release/0.9.0from
fix/mcp-park-waiting-tools-0.9.0
Oct 15, 2026
Merged

chenmingwei23 merged 1 commit into
release/0.9.0from
fix/mcp-park-waiting-tools-0.9.0

Conversation

@chenmingwei23

@chenmingwei23 chenmingwei23 commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: On the 0.9.0 line, one session's wait or spawn_sub_agents no longer makes every other session's kirocrew-core tool calls queue behind it.

This is a backport of #18841 (d9990a26fe, merged to main 2026-10-12) to release/0.9.0, cherry-picked with -x.

On 0.9.0 the pooled kirocrew-core backend has one tool worker thread. wait (up to 1800s), spawn_sub_agents (up to 7200s) and the spawn_run gate wait sleep on that thread. While one session waits, every other session's memory_recall, send_message, session_ledger_record and so on sits in the queue behind it (#17031). The backend keeps answering pings, so the gateway never recycles it.

Why it matters

Every session that shares the pooled kirocrew-core backend is hit. One babysit loop or one blocking subagent wave stalls the other chats' tools for minutes to hours, and new sessions can time out on session/new. 0.9.0 insiders who turn on shared MCP backends get this today, and #18841 only reaches them through the release branch.

Not a goal

  • Anything fix(mcp): park waiting tools off the worker so a wait stalls nobody #18841 itself does not do (see its own Not a goal): a second pooled process, session/new changes, concurrent dispatch of calls that compute, the Windows synchronous path.
  • The blocking ask_question from main. 0.9.0 ships the non-blocking ask, which never held the worker.
  • Other main changes to the same files that are not on 0.9.0 (per-task backend on spawn_run, the made_no_tool_calls note).

What changed (motivation → approach → change)

Symptom: one session's wait stalls every co-tenant's kirocrew-core call. Root cause: waiting handlers sleep on the backend's only worker thread. Change (from #18841, logic unchanged): those handlers return a DeferredTool. The stdio loop steps it on its own timer and an 8-thread step pool, and reports parked ids to the gateway (notifications/kirocrew/inflight) so its wedge detector neither recycles a healthy parked call nor ignores a stuck one.

release/0.9.0 is behind main in the touched files. Conflicts were resolved as follows:

Backwards compatibility

Compatible: tool names, schemas and results are unchanged for callers. wait and spawn_sub_agents return the same text, just without holding the worker. The new SPAWN_BATCH_MEMBERS_MAX (128) bound on spawn_run.tasks/agents and spawn_sub_agents.agents is twice the sub-agent ceiling of 64, the same as on main.

The diff removes the public mcp_shared.is_tool_cancelled() (as #18841 does on main). Its only readers on release/0.9.0 (git grep is_tool_cancelled b4cf148542 -- src) are the two blocking loops this PR replaces:

Reader: src/kiro_crew/mcp_tools/control.py:wait -- chat -- now returns _WaitStep (a DeferredTool); the dispatch loop drops a cancelled parked call; test_wait_tool_early_end.py
Reader: src/kiro_crew/mcp_tools/spawn.py:spawn_sub_agents -- subagent -- now returns _SubAgentsStep (a DeferredTool), cancelled the same way; test_mcp_core_spawn_sub_agents.py

Tests

From #18841, adapted to 0.9.0:

  • test_mcp_shared_deferred_tools.py (new): many parked calls hold no worker and all settle; per-call identity in each step; cancel before, during and after a step; parked overflow; pruned-install exit; the step pool's up-front threads and start failures.
  • test_mcp_gateway_inflight_report.py (new): the inflight capability handshake both ways and the gateway's handling of the report.
  • test_wait_tool_early_end.py, test_mcp_core_spawn_sub_agents.py, test_spawn_run_per_task_overrides.py, test_mcp_gateway_wedge_ping_gate.py and the other touched files: updated for the deferred return.

Results on this branch:

  • All 16 touched test files: 971 passed.
  • Every test_mcp*, test_spawn*, test_subagent*, test_wait*, test_gatewayd*, test_validation* file: 10664 passed, 42 skipped, 1 xfailed, 0 failed.
  • ruff check clean on every touched .py file.

Manual verification

N/A -- this is a cherry-pick of a change already verified on main; the tests above cover the parked and non-parked paths.

Related Issues

Backport of #18841. Refs #17031.

Pattern harvest

Not generalizable: this PR carries an existing fix to a release branch; the pattern harvest for the defect is in #18841.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributions are accepted under the project's Apache-2.0 license.

@chenmingwei23
chenmingwei23 requested review from a team and KiroCrewCaptain as code owners October 15, 2026 05:25
@chenmingwei23
chenmingwei23 requested review from bolichen97 and removed request for a team October 15, 2026 05:25
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 15, 2026
@github-actions

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision b9c8a7c66e79a9a59eb04e5b782c8a2cd4bbb3d5 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Advisory — does not block merge.

@github-actions

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of b9c8a7c66e79a9a59eb04e5b782c8a2cd4bbb3d5 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Check that the gateway still recycles a backend that is really stuck, given it now trusts the backend's own report of which calls are parked.

[DESIGN-REVIEWED] b9c8a7c

Description read

[DESCRIPTION-READ] 17cc9198b264c78981e163bb9542652630b47c3a16b0eaf6f4fb0df3e5c1c01c

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of b9c8a7c66e79a9a59eb04e5b782c8a2cd4bbb3d5 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

Before merge, confirm that every non-deferred kirocrew-core tool on 0.9.0 finishes well under the new 900s recycle ceiling. Any call that runs longer now recycles the shared backend.

Inventory (8 items) — 7 justified

Intent: stop one session's wait / spawn_sub_agents / spawn_run gate wait from holding the pooled backend's single tool worker, so other sessions' calls are no longer stuck behind it (#17031). This is a FIX, backported from #18841.

  1. wait, spawn_sub_agents and the spawn_run gate wait park off the worker and are stepped on the loop's own timer — justified
  2. The backend sends the gateway in-flight reports for its parked calls, through a new capability key and notification — justified
  3. A backend that sends those reports is now recycled when a call makes no progress for 900s, instead of after 3h — justified
  4. The wedge detector skips cancelled requests, so the shorter ceiling does not recycle a healthy backend — justified
  5. spawn_run.tasks/agents and spawn_sub_agents.agents are now capped at 128 members — justified
  6. Each retained refusal line is capped at 800 chars, and lost-run reports and keepalive POSTs time out after 5s — justified
  7. fix(mcp): answer every request with an id while a tool call runs #18979 comes in too: every request with an id gets an answer while a tool runs — rides along; the description says it does, and it keeps mcp_shared.py identical to main
  8. ask_question's return type now includes DeferredTool, but nothing in it returns one — justified

[FIRST-PRINCIPLES-REVIEWED] b9c8a7c

Description read

[DESCRIPTION-READ] 7cac2da6cc93b7de97c050f4a5d2b7bffb102413623d03f5e2fc951c7a9ce38a

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed b9c8a7c66e79a9a59eb04e5b782c8a2cd4bbb3d5 — this comment is updated in place on each push.

Review details

One advisory finding: the spec and a code comment describe an ask_question deferral that is not in this tree.

FINDING — docs/architecture/mcp.md:2906 — the spec says an ask step's slice is "validation.ASK_WAIT_SLICE_SECS (5s…)" and that the card queue budget "= 40s stays below the dashboard's 120s answer-retention grace"; the DEFERRED_STEP_WORKERS comment at src/kiro_crew/mcp_shared.py:314 repeats it. No such constant exists, and ask_question (mcp_tools/control.py:1563) always returns a plain _emit_directive string, never a DeferredTool. Anyone sizing the pool or reasoning about the 120s grace would be working from arithmetic about code that doesn't exist → Fix: delete the ask_question deferral, ASK_WAIT_SLICE_SECS and 40s-vs-120s claims from the spec paragraph and the DEFERRED_STEP_WORKERS comment, or land the ask deferral together with them.

[OPUS-REVIEWED] b9c8a7c

Verdict parsed from the review's SHA-scoped output markers for commit b9c8a7c66e79a9a59eb04e5b782c8a2cd4bbb3d5.

False positive or not applicable? A repository writer can comment:
/ai-review override fable b9c8a7c66e79a9a59eb04e5b782c8a2cd4bbb3d5: <one-sentence reason>

@github-actions

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of b9c8a7c66e79a9a59eb04e5b782c8a2cd4bbb3d5 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] b9c8a7c

False positive or not applicable? A repository writer can comment:
/ai-review override gpt b9c8a7c66e79a9a59eb04e5b782c8a2cd4bbb3d5: <one-sentence reason>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 15, 2026
@chenmingwei23
chenmingwei23 merged commit a9ec7fc into release/0.9.0 Oct 15, 2026
37 of 43 checks passed
@chenmingwei23
chenmingwei23 deleted the fix/mcp-park-waiting-tools-0.9.0 branch October 15, 2026 05:48
@github-actions github-actions Bot removed the readiness: action required A blocking check or review needs attention label Oct 15, 2026
bolichen97 pushed a commit that referenced this pull request Oct 15, 2026
Add one Notable fixes paragraph for the three fixes backported to release/0.9.0 after the section was written (#19189, #19309, #20631). Every existing line is unchanged.

Co-authored-by: kirocrewcaptain <kirocrewcaptain@amazon.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants