Skip to content

fix(mcp): run pooled kirocrew-core tool calls per-caller concurrently - #17761

Closed
billygerhard wants to merge 1 commit into
kirodotdev:mainfrom
billygerhard:fix/mcp-core-per-caller-concurrency-17031
Closed

billygerhard wants to merge 1 commit into
kirodotdev:mainfrom
billygerhard:fix/mcp-core-per-caller-concurrency-17031

Conversation

@billygerhard

@billygerhard billygerhard commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: One dashboard session's long tool call no longer stalls every other session's kirocrew-core calls.

The pooled kirocrew-core MCP backend — one process serves every dashboard session — dispatched tool calls strictly one at a time (single worker thread, FIFO _pending_calls). So one session's long cooperative tool (wait, up to 1800s; spawn_sub_agents) convoyed every other session's short call (memory_recall, resource_status, session_ledger_record, …) for minutes. The symptom an operator sees: repeated slow in-flight request … responsive (ping_age=…); not recycling warnings, cleared only by killing the backend pid. The gateway's wedge detector is ping-gated and the loop answers ping mid-call, so nothing recycled or scaled.

Why it matters

Every dashboard session shares the one pooled backend, so a single session parked on a 1800s wait makes everyone else's kirocrew-core tool calls hang for minutes to hours, and new sessions time out on session/new and come back untrusted. The only known workaround is manually killing the backend process. This is the one failure mode a single-worker pooled backend actually has, and the design guaranteed it would never self-heal.

Not a goal

What changed (motivation → approach → change)

Observed symptom: one long tool convoys all other sessions' calls. Root cause: the pooled backend's POSIX dispatch loop runs exactly one worker at a time, and the cooperative-cancel flag plus in-flight state are single module-global slots that assume that. The fix is to let independent callers run concurrently off the critical read loop.

  • Bounded concurrency, process-wide. The POSIX dispatch loop runs up to MCP_MAX_CONCURRENT_WORKERS (8) tool workers at once, scoped to server_name == "kirocrew-core" only (every other server sharing this dispatch loop — kirocrew-computer, kirocrew-cron, … — keeps the original one-worker-at-a-time behavior; kirocrew-computer specifically must never run two tool calls driving the same real mouse/keyboard at once). A tools/call with a free worker slot dispatches immediately; calls past the cap fall back to the existing FIFO _pending_calls queue and drain in order as slots free (nothing is dropped; overflow past PENDING_CALLS_MAX still gets an immediate busy error). The cap bounds total concurrency, not per-caller concurrency — see Backwards compatibility.
  • ContextVar cancel event (the linchpin). _thread_cancel_event is converted from a module global to a ContextVar[Optional[threading.Event]]. A threading.Thread starts each ContextVar at its default rather than inheriting the dispatch thread's context, so each worker's _run_tool sets its own event and is_tool_cancelled() — called from inside that worker thread by the cooperative tool it is running — reads that worker's event and no other's. A plain module slot would let one caller's cancel set or clear another's cooperative-cancel flag.
  • Per-request in-flight state. Result box, audit flag and cancel event move onto a _StdioWorker record keyed by _worker_key(req_id) (type + value, so JSON-RPC ids 1 and "1" do not collide), guarded by the loop's single _result_lock. One caller's result/cancel/late-cancel audit can no longer land against another caller's call. The reaper reads cancellation from each worker's own cancel_event rather than re-deriving it from the shared, string-keyed _cancelled_ids set, so a cancel for id 1 can never wrongly drop an uncancelled "1"'s result.
  • Caller identity (_CURRENT_CALLER / _CURRENT_TENANT_NONCE) is already a ContextVar and thread-safe for concurrency; this PR relies on that and updates the now-stale "dispatch is strictly sequential" comments.
  • Cancellation preserved per request. notifications/cancelled sets the matching worker's own event; a cancel against a still-queued request is still honored at dispatch via the bounded _cancelled_ids FIFO (with _live_request_ids() protection extended to every active worker).
  • Two new refusals, needed by concurrency itself. A tools/call is refused up front (never dispatched or queued) when: (1) its req_id, measured on repr(req_id) (covers a JSON-decoded container id too, not just a string), exceeds MAX_REQUEST_ID_BYTES (256) — concurrency means an id is now retained as a live dict key for the tool's whole run (up to 1800s), so an unbounded id is an unbounded per-request memory cost it did not use to be; (2) its req_id is already live (an existing _workers entry or a still-queued _pending_calls entry) — concurrent dispatch means a repeated id could otherwise silently overwrite the live worker record, leaving the first call's thread running uncounted against the cap and its eventual result orphaned.
  • Pruned-exit drain answers every call, including a drained-not-cancelled one. A pruned install's _drain_remaining_on_pruned_exit signals every in-flight worker's drain_event so a cooperative tool (wait/spawn_sub_agents) exits promptly instead of blocking the drain for its full duration. A worker that exits this way was never actually cancelled by its caller — it has no real result to hand back only because the process is shutting down — so it now gets the same retryable -32000 error the not-yet-started pruned-install path already answers queued calls with, instead of no response at all. A worker that was genuinely cancelled (notifications/cancelled arrived before the drain) still gets no response, as before.
  • Observability. The gateway's "slow in-flight request" warning (mcp_gateway/backend.py) now also logs the head-of-line tool name (tool=…), already captured on the pending record at forward time — the one piece of context that tells a legitimately long cooperative tool from a stuck one.

Backwards compatibility

Compatible: the public MCP protocol surface is unchanged — same requests, same responses, same JSON-RPC id semantics. is_tool_cancelled() keeps the same signature and semantics (it now reads a ContextVar instead of a module global); all in-tree readers call it through the public function, not the global. Windows behavior is byte-for-byte unchanged (the dispatch loop there stays fully synchronous; the concurrency cap and all of the above apply only to the POSIX worker-thread path). No field is rejected, renamed or removed; nothing a caller does today stops working.

Response ordering for a single caller's own concurrent calls is no longer FIFO. _slot_free() bounds concurrency process-wide (len(_workers) < MCP_MAX_CONCURRENT_WORKERS for kirocrew-core), not per caller_key — it has no notion of which calls share a caller. If one session sends call A then call B back to back and both land on free slots, they now run concurrently and can complete (and respond) in either order, where before dispatch was strictly one-at-a-time so A always finished before B started. This PR does not serialize a caller's own calls relative to each other; doing so would reintroduce a scoped-down version of the convoy this PR removes (one caller's own slow call blocking its own next call) for the case of a client that intentionally issues independent concurrent tool calls. kiro-cli's MCP client issues one tool call per turn and awaits its response before issuing the next, so this does not change observed behavior for that client today, but a client that pipelines multiple calls on one connection must not rely on completion order matching send order.

Removes nothing: no capability, record, key or public function is deleted or narrowed. The module-global _thread_cancel_event becomes a ContextVar, but it is internal state; its only readers (is_tool_cancelled() callers in spawn.py/control.py, and tests) go through the public function or are updated to the ContextVar API in this PR.

Tests

In test/test_mcp_shared.py:

  • test_a_slow_caller_does_not_convoy_a_different_callers_fast_call — caller A holds a slow tool open while caller B sends a fast call through the loop; the fast call returns before the slow one is released (the regression this PR fixes).
  • test_a_cancel_for_one_concurrent_worker_does_not_trip_another — two workers run concurrently, one is cancelled; the cancelled worker's is_tool_cancelled() reads True and the other's stays False (pins the ContextVar conversion; verified it fails against a shared-global mutation).
  • test_non_core_servers_keep_the_original_one_worker_at_a_time_behavior — a server other than kirocrew-core sharing this dispatch loop never gets the multi-worker cap.
  • Oversized-id and duplicate-in-flight-id refusals, including a container (dict) id, each with a mutation-probed regression test.
  • test_a_cooperative_worker_is_drained_promptly_on_the_pruned_exit — a drained-but-not-cancelled worker on a pruned exit gets the retryable -32000 error, not silence.
  • test_a_genuinely_cancelled_worker_still_gets_no_response_on_pruned_exit — a worker genuinely cancelled before the drain reaches it still gets no response, distinguishing the two outcomes.

Existing busy-queue / overflow / cancel-of-queued / prune-of-queued tests are updated to saturate the worker cap first, so they still exercise the FIFO queue (which now only engages once the pool is full). test_cancel_event_check and test/test_mcp_core_audit_e.py are updated to use the ContextVar API.

Manual verification

N/A — unit coverage is sufficient: the convoy regression, the per-worker cancel isolation, the two new refusals and the drain-vs-cancel distinction are all pinned by tests, and each production fix has a mutation probe confirming its test fails without it.

Related Issues

Fixes #17031

Pattern harvest

Rule candidate: review-prompt
Pattern: a module-global "current in-flight" slot (cancel flag, caller, request id) that is safe only because dispatch is single-worker — introducing concurrency silently corrupts it unless the slot is first moved to a ContextVar or a per-request record. A second instance of the same class: a drain/exit signal that reuses a cooperative tool's existing cancel-check path (is_tool_cancelled()) must be distinguishable from a REAL cancel at the point the exception is handled, or "the process is shutting down" and "the caller asked to stop" collapse into the same (wrong, for the first case) outcome.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@billygerhard
billygerhard requested a review from a team as a code owner October 7, 2026 14:26
@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: action required A blocking check or review needs attention labels Oct 7, 2026
@billygerhard
billygerhard force-pushed the fix/mcp-core-per-caller-concurrency-17031 branch from 63705be to 57721fd Compare October 7, 2026 15:33
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 7, 2026
@billygerhard
billygerhard force-pushed the fix/mcp-core-per-caller-concurrency-17031 branch from 57721fd to 42d253e Compare October 7, 2026 15:53
@github-actions

github-actions Bot commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5, fork) — 🟡 CONCERNS

Premise-level review of fc541474ae74088ddb2451fe98c2b742b741bef9 via the fork AI-review pipeline — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: CONCERNS

The 256-byte request-id refusal rests on a false premise: the base already keeps ids for a call's whole run, and tool arguments stay unbounded anyway.

Not justified as shipped

  • Item 6 — inherited: the description says a long id "is now an unbounded per-request memory cost it did not use to be". Base already keeps _current_req_id for the worker's whole run, plus up to PENDING_CALLS_MAX (32) queued ids, each bounded only by MAX_CONTENT_LENGTH_BYTES (64 MiB). Concurrency raises that from 33 to 40 retained requests, and each worker's tool_args stays just as unbounded.
    Subtraction: delete MAX_REQUEST_ID_BYTES, the repr(req_id) size check in _route_tool_call, and the rejected_oversized_id audit.
    Clears when: the check is deleted, or a measured cost shows the id, and not the request's arguments, is what grows with the new concurrency.

What this change ships

Inventory (8 items) — 6 justified

Intent: one session's long kirocrew-core tool call should stop stalling every other session's calls. FIX.

  1. Up to 8 kirocrew-core tool calls now run at once; other servers still run one at a time — justified
  2. A cancel now stops only its own call; cancel and drain flags become per-worker ContextVars — justified
  3. The directive vouch record becomes per-worker, so one session can't forge another's provenance — justified
  4. Concurrent responses from one caller can now arrive out of order — justified
  5. A tools/call reusing an id that is still running or queued is refused — justified
  6. A tools/call whose id is over 256 bytes is refused — inherited
  7. On a pruned-install exit, running workers are drained; a drained call that was never cancelled gets a retryable -32000 — justified
  8. The slow-request warning now logs the tool name — rides along

[FIRST-PRINCIPLES-REVIEWED] fc54147

Description read

[DESCRIPTION-READ] 87cdf753a77532187e5aea1ae40e379f1c74dcb4fcdb9a3b12b0961f827f6601

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5, fork) — 🟡 CONCERNS

Design-level review of fc541474ae74088ddb2451fe98c2b742b741bef9 via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: CONCERNS

Eight slots shared by everyone still let long wait/spawn_sub_agents calls fill the pool and bring the convoy back; the per-caller key is stored but never used.

Watch

  • Saturation brings the convoy back. _slot_free() only checks len(_workers) < MCP_MAX_CONCURRENT_WORKERS (8). worker.caller_key is recorded on every worker but nothing reads it. If eight sessions or sub-agents each sit in an 1800s wait, every other caller's memory_recall goes back into the FIFO _pending_calls queue for minutes. That is the same failure the Goal line says this PR ends. The comment calls 8 "a handful of slots already guarantees" without measuring a real tool mix, and gateway scaling is deferred to a follow-up.
    Clears when: either a per-caller_key cap (or a separate lane for the cooperative long tools) stops one class of call from filling every slot, or a test or measurement shows that N concurrent long tools still leave the next short call from a different caller answered promptly.
  • The compatibility check is capped. ## Backwards compatibility is the last section before the [description TRUNCATED notice, so its reader list may be cut off. The two new refusals (an oversized id and an id already in flight) refuse calls the server used to accept. The gateway rewrites ids into its own forward ids, so I found no in-tree reader that breaks, but I could not compare against the full list.
    Clears when: the section fits under the 8000-byte cut and either names the gateway's forward-id path (mcp_gateway/backend.py) as a reader with the reason it never repeats an id, or states Removes nothing.

Suggestions

  • Turn concurrency on when the process is a pooled backend (install_liveness.POOLED_BACKEND_ENV) rather than when server_name == "kirocrew-core". The pooled property is the actual reason concurrency is needed, and a server rename then can't silently switch it off or on.

[DESIGN-REVIEWED] fc54147

Description read

[DESCRIPTION-READ] eaca47e8f0dd1d177ae4cf3dd075f894d5da493c0290311c7345987ef9231526

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 7, 2026
@github-actions

github-actions Bot commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review (fork) — ✅ no blocking findings

Reviewed fc541474ae74088ddb2451fe98c2b742b741bef9 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] fc54147

@github-actions

github-actions Bot commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review (fork) — ✅ no blocking findings

Reviewed fc541474ae74088ddb2451fe98c2b742b741bef9 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings that block the merge.

FINDING — src/kiro_crew/mcp_shared.py:1849 — When a pruned install drains a spawn_sub_agents call that is already polling at spawn.py:1486, the call raises ToolCancelled, the drain path sets drained_only = not cancel_evt.is_set() and boxes "...stopped this call early; retry it". By then /api/spawn has already started the children, so a caller that obeys the message starts a duplicate set. On base, a call already in flight was never interrupted by the prune and ran to completion. → Fix: for a drained exit after side effects, don't send the "retry it" message (have spawn return its started ids as a normal result, or skip the drain signal for spawn_sub_agents).

[OPUS-REVIEWED] fc54147

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 7, 2026
@billygerhard
billygerhard force-pushed the fix/mcp-core-per-caller-concurrency-17031 branch 2 times, most recently from 6721c37 to 370fbb5 Compare October 7, 2026 17:47
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 7, 2026
@billygerhard
billygerhard force-pushed the fix/mcp-core-per-caller-concurrency-17031 branch from 370fbb5 to f0ea115 Compare October 7, 2026 19:43
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 7, 2026
@billygerhard
billygerhard force-pushed the fix/mcp-core-per-caller-concurrency-17031 branch from f0ea115 to 1372323 Compare October 7, 2026 20:46
@billygerhard
billygerhard force-pushed the fix/mcp-core-per-caller-concurrency-17031 branch from 361ee5e to debda36 Compare October 8, 2026 08:12
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 8, 2026
@billygerhard
billygerhard force-pushed the fix/mcp-core-per-caller-concurrency-17031 branch 2 times, most recently from ea54aa7 to a9bba08 Compare October 8, 2026 08:59
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running and removed readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention labels Oct 8, 2026
@billygerhard
billygerhard force-pushed the fix/mcp-core-per-caller-concurrency-17031 branch from a9bba08 to db3db64 Compare October 8, 2026 09:14
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running and removed readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention labels Oct 8, 2026
@billygerhard
billygerhard force-pushed the fix/mcp-core-per-caller-concurrency-17031 branch from db3db64 to af7c082 Compare October 8, 2026 09:47
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running and removed readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention labels Oct 8, 2026
@billygerhard
billygerhard force-pushed the fix/mcp-core-per-caller-concurrency-17031 branch from af7c082 to 4595b92 Compare October 8, 2026 10:19

@iamwhatever iamwhatever left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This fixes one session's long wait / spawn_sub_agents call blocking every other session's kirocrew-core calls on the pooled backend. It does so by running up to 8 workers at once, with a ContextVar cancel event for each worker, so the convoy's root cause is addressed. However, a new path drops a response, which is a regression. _drain_remaining_on_pruned_exit (mcp_shared.py:2524) sets each worker's drain_event, and is_tool_cancelled() reads that event. So a running wait raises ToolCancelled, and the handler at mcp_shared.py:1818 returns with an empty result_box, which means the caller gets no response at all. Before this PR the call would have finished. The drain's own docstring promises the opposite ("must still get its real result"). Suggested fix: answer a drain-cancelled request with the retryable -32000 pruned-install error, or keep the loop answering pings and refusing new calls until the in-flight workers finish on their own, and add a test that drives a cooperative tool through a pruned exit and asserts a response. The Design and First Principles CONCERNS also have no disposition. _slot_free() caps workers for the whole process and does not serialize per caller_key, so one session's own calls now overlap, which contradicts the "same ordering guarantees per caller" claim. The PR also adds two refusals its description does not list: duplicate in-flight ids and ids over 256 bytes. Please fix the drain path, then serialize per caller or correct the description, and answer each item. It also changes session_directive._VOUCHED provenance, so a maintainer needs to review it after the fix. — crew-pr-reviewer

The pooled kirocrew-core MCP backend (one process serves every dashboard
session) dispatched tool calls strictly one at a time, so a single session's
long cooperative tool (wait, up to 1800s; spawn_sub_agents) convoyed every
other session's short call (memory_recall, resource_status, ...) for minutes.
The ping-gated wedge detector stays responsive mid-call, so nothing recycled.

Let independent callers run off the critical read loop in their own worker
threads, bounded by MCP_MAX_CONCURRENT_WORKERS (8); calls past the cap fall
back to the existing FIFO _pending_calls queue. The linchpin is converting the
module-global _thread_cancel_event to a ContextVar so is_tool_cancelled() reads
the current worker thread's own event instead of a slot every worker clobbers.
Per-request in-flight state (result box, audit flag, cancel event) moves onto a
_StdioWorker record keyed by req_id, so one caller's result/cancel/audit cannot
land against another's. Caller identity is already a ContextVar and is
thread-safe for concurrency. Windows stays synchronous (select cannot poll
stdin there).

Also surfaces the head-of-line tool name in the gateway's "slow in-flight
request" warning, so an operator can tell a legitimately long cooperative tool
from a stuck one.

Tests: a slow cooperative tool from one caller does not block a different
caller's fast call; is_tool_cancelled() is per-worker (a cancel for one
concurrent worker does not trip another's). Existing busy-queue, overflow,
cancel-of-queued and prune-of-queued tests updated to saturate the worker cap
so they still exercise the FIFO queue under concurrency.

Fixes kirodotdev#17031
@billygerhard

Copy link
Copy Markdown
Contributor Author

Thanks for catching the drain regression -- that was real. Fixed all three items on fc541474a:

  1. Drain-silently-drops-a-response (blocking, fixed). _run_tool's ToolCancelled handler now distinguishes a drain-only exit (cancel_evt not set -- nobody asked to cancel this call, it just exited early for the pruned-install shutdown) from a real cancel. A drained-not-cancelled worker now gets the same retryable -32000 error the not-yet-started pruned-install path already answers queued calls with, instead of no response at all. A genuinely cancelled worker (cancel arrived before the drain) still gets no response, as before -- added a test (test_a_genuinely_cancelled_worker_still_gets_no_response_on_pruned_exit) alongside the updated drain test to pin the distinction, both mutation-probed.

  2. Per-caller ordering claim (addressed by correcting the description, not by serializing). You're right that _slot_free() caps concurrency process-wide, not per caller_key -- a single caller's own back-to-back calls can now land on separate free slots and complete in either order, where before dispatch was strictly one-at-a-time. I looked at adding per-caller serialization and decided against it: it would reintroduce a scoped-down version of the exact convoy this PR removes (one caller's own slow call blocking its own next call), for the case of a client that intentionally pipelines independent concurrent calls. kiro-cli's actual MCP client issues one tool call per turn and awaits the response before the next, so this doesn't change observed behavior for it today. I corrected the "same ordering guarantees per caller" line in Backwards compatibility to state the real guarantee (process-wide bound, not per-caller FIFO) and spelled out why, plus added a "Not a goal" bullet.

  3. Undescribed refusals. Added a dedicated "What changed" bullet documenting both the oversized-id (MAX_REQUEST_ID_BYTES, measured via repr() so a container id can't bypass it) and duplicate-in-flight-id refusals, with the concurrency-specific reason each exists (an id is now retained as a live dict key for the tool's whole run, and a duplicate id could silently overwrite a live worker record).

Full description rewrite is in the PR body now. Re-ran the full local gate set (flake8, black --target-version py310, isort, mypy, pytest, comment-history) after the fix -- all clean, including the two new regression tests.

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for taking #17031 on -- the diagnosis in the body is exactly right, and the per-worker ContextVar cancel plumbing is clean. Requesting changes on the shape of the fix rather than on a defect in it, and to say out loud that we are landing a companion change on the same loop so the two do not collide.

Why 8 workers is a mitigation, not the fix. The convoy exists because wait and spawn_sub_agents spend their entire duration asleep on the worker (time.sleep in control.py, a 2s poll loop in spawn.py) while doing nothing but a periodic keepalive. A worker pool moves the threshold from 1 to 8. A babysit fleet with nine sessions parked on wait -- an ordinary afternoon on this repo -- reproduces #17031 exactly, and MCP_MAX_CONCURRENT_WORKERS then becomes the next knob somebody raises.

What we are doing instead (companion PR, in progress on fix/mcp-deferred-long-tools, closes #17031). The tool that waits should not hold a thread:

  • run_mcp_stdio_loop gains a DeferredTool protocol: a handler may return a deferred object whose short step() the dispatch loop runs from its own select timer. The worker frees the moment the handler returns, so a thousand sessions can be parked on wait and the one worker stays idle.
  • wait and spawn_sub_agents become deferred (setup in the worker, the sleep/poll as step()s, cancel via the loop). No behaviour change for the caller.
  • The backend reports its in-flight deferred ids to the gateway (notifications/kirocrew/inflight, capability-gated in both directions in initialize, so an older peer never sees the frame). The wedge detector refreshes those requests' age on each report, which lets a progress-aware backend use a 900s ceiling instead of SUBAGENT_TIMEOUT_SECS + 300 (3h05m) -- a genuinely hung worker call then recycles in minutes, which is the self-heal the issue asks for and a worker pool cannot provide.

How the two relate. They are complementary: once the by-design long calls are off the worker, extra workers only buy headroom against a pathological hung call (today's longest legitimate synchronous call is pod_up at 210s). If you want to keep this PR, please hold it until the companion lands and rebase onto it -- two points will need a look then: the deferred registry must stay on the dispatch thread (never a worker), and _drain_remaining_on_pruned_exit needs to also settle deferred entries, not only workers.

One unrelated note: the Integration red here (test/integration/test_owner_only_boot.py:81, an autonudge spool file created 0o644) touches nothing this diff changes; worth a rerun on a fresh rebase before chasing it.

I will link the companion PR here as soon as it is open.

@billygerhard

Copy link
Copy Markdown
Contributor Author

Thanks @bolichen97 -- that makes sense, and the deferred-tool approach is clearly the right fix: taking the by-design long calls (wait, spawn_sub_agents) off the worker entirely addresses the convoy at its root, where a worker pool only raises the threshold. I agree a nine-session babysit fleet parked on wait would reproduce #17031 straight through 8 workers.

I'll hold this PR and rebase onto fix/mcp-deferred-long-tools once it lands, rather than progressing it independently. When I rebase I'll make sure (1) the deferred registry stays on the dispatch thread and is never handed to a worker, and (2) _drain_remaining_on_pruned_exit settles deferred entries as well as workers -- the drain-path response-drop fix in this PR already distinguishes a drain-only exit from a real cancel, so extending it to deferred entries should slot in cleanly.

No rush on the companion -- just link it here when it's open and I'll take it from there. Leaving the inherited test_owner_only_boot.py:81 red alone until the rebase, as you suggested.

@bolichen97

Copy link
Copy Markdown
Contributor

The companion change is up: PR #18841 (closes #17031).

What it does, in one line: wait, spawn_sub_agents and ask_question return a DeferredTool and the dispatch loop times their steps itself, so a parked call holds no worker; the backend reports its parked ids to the gateway, which holds a reporting backend to a 900s ceiling instead of 3h.

Two spots to look at when rebasing this PR onto it, as noted above: the deferred registry (_deferred, _StepPool) must stay on the dispatch thread, and your _drain_remaining_on_pruned_exit should settle deferred entries the way _refuse_parked_on_pruned_exit does (finish a step in flight, cancel a queued one, refuse the rest).

@chenmingwei23 chenmingwei23 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is against the original spirit of mcp pooling. The pool was developed to share process across multiple sessions, forking off a a backend mcp process means we are reverting it back thus the memory pressure comes back

@billygerhard

Copy link
Copy Markdown
Contributor Author

Closing this in favor of the merged companion #18841, which fixes #17031 at the root by making wait/spawn_sub_agents/ask_question deferred so a parked call holds no worker at all.

Thanks @bolichen97 and @chenmingwei23 -- the point lands: the worker pool here was a mitigation that raises the convoy threshold rather than removing it, and it cuts against the one-process-shared-across-sessions goal of the pool. The deferred-tool approach is the right shape and it is already on main, so there is nothing unique left for this PR to carry.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor) merge conflict Branch has merge conflicts with its base — author must resolve before merge

Projects

None yet

4 participants