Repository navigation
fix(mcp): run pooled kirocrew-core tool calls per-caller concurrently - #17761
billygerhard wants to merge 1 commit into
Conversation
63705be to
57721fd
Compare
57721fd to
42d253e
Compare
First Principles Review (Opus 5.5, fork) — 🟡 CONCERNSPremise-level review of First-Principles-Verdict: CONCERNS The 256-byte request-id refusal rests on a false premise: the base already keeps ids for a call's whole run, and tool arguments stay unbounded anyway. Not justified as shipped
What this change shipsInventory (8 items) — 6 justifiedIntent: one session's long
[FIRST-PRINCIPLES-REVIEWED] fc54147 Description read[DESCRIPTION-READ] 87cdf753a77532187e5aea1ae40e379f1c74dcb4fcdb9a3b12b0961f827f6601 That sha256 names the PR description this verdict read. Recompute it with |
Design Review (Opus 5.5, fork) — 🟡 CONCERNSDesign-level review of Design-Verdict: CONCERNS Eight slots shared by everyone still let long Watch
Suggestions
[DESIGN-REVIEWED] fc54147 Description read[DESCRIPTION-READ] eaca47e8f0dd1d177ae4cf3dd075f894d5da493c0290311c7345987ef9231526 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
GPT 6.1 Review (fork) — ✅ no blocking findingsReviewed Review detailsNo findings. |
Opus 5.5 Review (fork) — ✅ no blocking findingsReviewed Review detailsNo findings that block the merge. FINDING — src/kiro_crew/mcp_shared.py:1849 — When a pruned install drains a [OPUS-REVIEWED] fc54147 |
6721c37 to
370fbb5
Compare
370fbb5 to
f0ea115
Compare
f0ea115 to
1372323
Compare
361ee5e to
debda36
Compare
ea54aa7 to
a9bba08
Compare
a9bba08 to
db3db64
Compare
db3db64 to
af7c082
Compare
af7c082 to
4595b92
Compare
iamwhatever
left a comment
There was a problem hiding this comment.
This fixes one session's long wait / spawn_sub_agents call blocking every other session's kirocrew-core calls on the pooled backend. It does so by running up to 8 workers at once, with a ContextVar cancel event for each worker, so the convoy's root cause is addressed. However, a new path drops a response, which is a regression. _drain_remaining_on_pruned_exit (mcp_shared.py:2524) sets each worker's drain_event, and is_tool_cancelled() reads that event. So a running wait raises ToolCancelled, and the handler at mcp_shared.py:1818 returns with an empty result_box, which means the caller gets no response at all. Before this PR the call would have finished. The drain's own docstring promises the opposite ("must still get its real result"). Suggested fix: answer a drain-cancelled request with the retryable -32000 pruned-install error, or keep the loop answering pings and refusing new calls until the in-flight workers finish on their own, and add a test that drives a cooperative tool through a pruned exit and asserts a response. The Design and First Principles CONCERNS also have no disposition. _slot_free() caps workers for the whole process and does not serialize per caller_key, so one session's own calls now overlap, which contradicts the "same ordering guarantees per caller" claim. The PR also adds two refusals its description does not list: duplicate in-flight ids and ids over 256 bytes. Please fix the drain path, then serialize per caller or correct the description, and answer each item. It also changes session_directive._VOUCHED provenance, so a maintainer needs to review it after the fix. — crew-pr-reviewer
The pooled kirocrew-core MCP backend (one process serves every dashboard session) dispatched tool calls strictly one at a time, so a single session's long cooperative tool (wait, up to 1800s; spawn_sub_agents) convoyed every other session's short call (memory_recall, resource_status, ...) for minutes. The ping-gated wedge detector stays responsive mid-call, so nothing recycled. Let independent callers run off the critical read loop in their own worker threads, bounded by MCP_MAX_CONCURRENT_WORKERS (8); calls past the cap fall back to the existing FIFO _pending_calls queue. The linchpin is converting the module-global _thread_cancel_event to a ContextVar so is_tool_cancelled() reads the current worker thread's own event instead of a slot every worker clobbers. Per-request in-flight state (result box, audit flag, cancel event) moves onto a _StdioWorker record keyed by req_id, so one caller's result/cancel/audit cannot land against another's. Caller identity is already a ContextVar and is thread-safe for concurrency. Windows stays synchronous (select cannot poll stdin there). Also surfaces the head-of-line tool name in the gateway's "slow in-flight request" warning, so an operator can tell a legitimately long cooperative tool from a stuck one. Tests: a slow cooperative tool from one caller does not block a different caller's fast call; is_tool_cancelled() is per-worker (a cancel for one concurrent worker does not trip another's). Existing busy-queue, overflow, cancel-of-queued and prune-of-queued tests updated to saturate the worker cap so they still exercise the FIFO queue under concurrency. Fixes kirodotdev#17031
|
Thanks for catching the drain regression -- that was real. Fixed all three items on
Full description rewrite is in the PR body now. Re-ran the full local gate set (flake8, black --target-version py310, isort, mypy, pytest, comment-history) after the fix -- all clean, including the two new regression tests. |
bolichen97
left a comment
There was a problem hiding this comment.
Thanks for taking #17031 on -- the diagnosis in the body is exactly right, and the per-worker ContextVar cancel plumbing is clean. Requesting changes on the shape of the fix rather than on a defect in it, and to say out loud that we are landing a companion change on the same loop so the two do not collide.
Why 8 workers is a mitigation, not the fix. The convoy exists because wait and spawn_sub_agents spend their entire duration asleep on the worker (time.sleep in control.py, a 2s poll loop in spawn.py) while doing nothing but a periodic keepalive. A worker pool moves the threshold from 1 to 8. A babysit fleet with nine sessions parked on wait -- an ordinary afternoon on this repo -- reproduces #17031 exactly, and MCP_MAX_CONCURRENT_WORKERS then becomes the next knob somebody raises.
What we are doing instead (companion PR, in progress on fix/mcp-deferred-long-tools, closes #17031). The tool that waits should not hold a thread:
run_mcp_stdio_loopgains aDeferredToolprotocol: a handler may return a deferred object whose shortstep()the dispatch loop runs from its ownselecttimer. The worker frees the moment the handler returns, so a thousand sessions can be parked onwaitand the one worker stays idle.waitandspawn_sub_agentsbecome deferred (setup in the worker, the sleep/poll asstep()s, cancel via the loop). No behaviour change for the caller.- The backend reports its in-flight deferred ids to the gateway (
notifications/kirocrew/inflight, capability-gated in both directions ininitialize, so an older peer never sees the frame). The wedge detector refreshes those requests' age on each report, which lets a progress-aware backend use a 900s ceiling instead ofSUBAGENT_TIMEOUT_SECS + 300(3h05m) -- a genuinely hung worker call then recycles in minutes, which is the self-heal the issue asks for and a worker pool cannot provide.
How the two relate. They are complementary: once the by-design long calls are off the worker, extra workers only buy headroom against a pathological hung call (today's longest legitimate synchronous call is pod_up at 210s). If you want to keep this PR, please hold it until the companion lands and rebase onto it -- two points will need a look then: the deferred registry must stay on the dispatch thread (never a worker), and _drain_remaining_on_pruned_exit needs to also settle deferred entries, not only workers.
One unrelated note: the Integration red here (test/integration/test_owner_only_boot.py:81, an autonudge spool file created 0o644) touches nothing this diff changes; worth a rerun on a fresh rebase before chasing it.
I will link the companion PR here as soon as it is open.
|
Thanks @bolichen97 -- that makes sense, and the deferred-tool approach is clearly the right fix: taking the by-design long calls ( I'll hold this PR and rebase onto No rush on the companion -- just link it here when it's open and I'll take it from there. Leaving the inherited |
|
The companion change is up: PR #18841 (closes #17031). What it does, in one line: Two spots to look at when rebasing this PR onto it, as noted above: the deferred registry ( |
chenmingwei23
left a comment
There was a problem hiding this comment.
This is against the original spirit of mcp pooling. The pool was developed to share process across multiple sessions, forking off a a backend mcp process means we are reverting it back thus the memory pressure comes back
|
Closing this in favor of the merged companion #18841, which fixes #17031 at the root by making Thanks @bolichen97 and @chenmingwei23 -- the point lands: the worker pool here was a mitigation that raises the convoy threshold rather than removing it, and it cuts against the one-process-shared-across-sessions goal of the pool. The deferred-tool approach is the right shape and it is already on main, so there is nothing unique left for this PR to carry. |
Problem / Motivation
Goal: One dashboard session's long tool call no longer stalls every other session's
kirocrew-corecalls.The pooled
kirocrew-coreMCP backend — one process serves every dashboard session — dispatched tool calls strictly one at a time (single worker thread, FIFO_pending_calls). So one session's long cooperative tool (wait, up to 1800s;spawn_sub_agents) convoyed every other session's short call (memory_recall,resource_status,session_ledger_record, …) for minutes. The symptom an operator sees: repeatedslow in-flight request … responsive (ping_age=…); not recyclingwarnings, cleared only by killing the backend pid. The gateway's wedge detector is ping-gated and the loop answerspingmid-call, so nothing recycled or scaled.Why it matters
Every dashboard session shares the one pooled backend, so a single session parked on a 1800s
waitmakes everyone else'skirocrew-coretool calls hang for minutes to hours, and new sessions time out onsession/newand come back untrusted. The only known workaround is manually killing the backend process. This is the one failure mode a single-worker pooled backend actually has, and the design guaranteed it would never self-heal.Not a goal
kirocrew-corebackend under saturation) — tracked as a follow-up; this PR removes only the per-process convoy.session/newso it does not depend on the pooled backend's queue — follow-up.select.selectcannot poll stdin there); that is a separate problem (mcp-gateway: on Windows, a kirocrew-core tool call over ~5 minutes gets the shared backend recycled, because the stdio loop cannot answer heartbeat pings mid-call #14852).What changed (motivation → approach → change)
Observed symptom: one long tool convoys all other sessions' calls. Root cause: the pooled backend's POSIX dispatch loop runs exactly one worker at a time, and the cooperative-cancel flag plus in-flight state are single module-global slots that assume that. The fix is to let independent callers run concurrently off the critical read loop.
MCP_MAX_CONCURRENT_WORKERS(8) tool workers at once, scoped toserver_name == "kirocrew-core"only (every other server sharing this dispatch loop —kirocrew-computer,kirocrew-cron, … — keeps the original one-worker-at-a-time behavior;kirocrew-computerspecifically must never run two tool calls driving the same real mouse/keyboard at once). Atools/callwith a free worker slot dispatches immediately; calls past the cap fall back to the existing FIFO_pending_callsqueue and drain in order as slots free (nothing is dropped; overflow pastPENDING_CALLS_MAXstill gets an immediate busy error). The cap bounds total concurrency, not per-caller concurrency — see Backwards compatibility._thread_cancel_eventis converted from a module global to aContextVar[Optional[threading.Event]]. Athreading.Threadstarts each ContextVar at its default rather than inheriting the dispatch thread's context, so each worker's_run_toolsets its own event andis_tool_cancelled()— called from inside that worker thread by the cooperative tool it is running — reads that worker's event and no other's. A plain module slot would let one caller's cancel set or clear another's cooperative-cancel flag._StdioWorkerrecord keyed by_worker_key(req_id)(type + value, so JSON-RPC ids1and"1"do not collide), guarded by the loop's single_result_lock. One caller's result/cancel/late-cancel audit can no longer land against another caller's call. The reaper reads cancellation from each worker's owncancel_eventrather than re-deriving it from the shared, string-keyed_cancelled_idsset, so a cancel for id1can never wrongly drop an uncancelled"1"'s result._CURRENT_CALLER/_CURRENT_TENANT_NONCE) is already aContextVarand thread-safe for concurrency; this PR relies on that and updates the now-stale "dispatch is strictly sequential" comments.notifications/cancelledsets the matching worker's own event; a cancel against a still-queued request is still honored at dispatch via the bounded_cancelled_idsFIFO (with_live_request_ids()protection extended to every active worker).tools/callis refused up front (never dispatched or queued) when: (1) itsreq_id, measured onrepr(req_id)(covers a JSON-decoded container id too, not just a string), exceedsMAX_REQUEST_ID_BYTES(256) — concurrency means an id is now retained as a live dict key for the tool's whole run (up to 1800s), so an unbounded id is an unbounded per-request memory cost it did not use to be; (2) itsreq_idis already live (an existing_workersentry or a still-queued_pending_callsentry) — concurrent dispatch means a repeated id could otherwise silently overwrite the live worker record, leaving the first call's thread running uncounted against the cap and its eventual result orphaned._drain_remaining_on_pruned_exitsignals every in-flight worker'sdrain_eventso a cooperative tool (wait/spawn_sub_agents) exits promptly instead of blocking the drain for its full duration. A worker that exits this way was never actually cancelled by its caller — it has no real result to hand back only because the process is shutting down — so it now gets the same retryable-32000error the not-yet-started pruned-install path already answers queued calls with, instead of no response at all. A worker that was genuinely cancelled (notifications/cancelledarrived before the drain) still gets no response, as before.mcp_gateway/backend.py) now also logs the head-of-line tool name (tool=…), already captured on the pending record at forward time — the one piece of context that tells a legitimately long cooperative tool from a stuck one.Backwards compatibility
Compatible: the public MCP protocol surface is unchanged — same requests, same responses, same JSON-RPC id semantics.
is_tool_cancelled()keeps the same signature and semantics (it now reads a ContextVar instead of a module global); all in-tree readers call it through the public function, not the global. Windows behavior is byte-for-byte unchanged (the dispatch loop there stays fully synchronous; the concurrency cap and all of the above apply only to the POSIX worker-thread path). No field is rejected, renamed or removed; nothing a caller does today stops working.Response ordering for a single caller's own concurrent calls is no longer FIFO.
_slot_free()bounds concurrency process-wide (len(_workers) < MCP_MAX_CONCURRENT_WORKERSforkirocrew-core), not percaller_key— it has no notion of which calls share a caller. If one session sends call A then call B back to back and both land on free slots, they now run concurrently and can complete (and respond) in either order, where before dispatch was strictly one-at-a-time so A always finished before B started. This PR does not serialize a caller's own calls relative to each other; doing so would reintroduce a scoped-down version of the convoy this PR removes (one caller's own slow call blocking its own next call) for the case of a client that intentionally issues independent concurrent tool calls. kiro-cli's MCP client issues one tool call per turn and awaits its response before issuing the next, so this does not change observed behavior for that client today, but a client that pipelines multiple calls on one connection must not rely on completion order matching send order.Removes nothing: no capability, record, key or public function is deleted or narrowed. The module-global
_thread_cancel_eventbecomes a ContextVar, but it is internal state; its only readers (is_tool_cancelled()callers inspawn.py/control.py, and tests) go through the public function or are updated to the ContextVar API in this PR.Tests
In
test/test_mcp_shared.py:test_a_slow_caller_does_not_convoy_a_different_callers_fast_call— caller A holds a slow tool open while caller B sends a fast call through the loop; the fast call returns before the slow one is released (the regression this PR fixes).test_a_cancel_for_one_concurrent_worker_does_not_trip_another— two workers run concurrently, one is cancelled; the cancelled worker'sis_tool_cancelled()reads True and the other's stays False (pins the ContextVar conversion; verified it fails against a shared-global mutation).test_non_core_servers_keep_the_original_one_worker_at_a_time_behavior— a server other thankirocrew-coresharing this dispatch loop never gets the multi-worker cap.test_a_cooperative_worker_is_drained_promptly_on_the_pruned_exit— a drained-but-not-cancelled worker on a pruned exit gets the retryable-32000error, not silence.test_a_genuinely_cancelled_worker_still_gets_no_response_on_pruned_exit— a worker genuinely cancelled before the drain reaches it still gets no response, distinguishing the two outcomes.Existing busy-queue / overflow / cancel-of-queued / prune-of-queued tests are updated to saturate the worker cap first, so they still exercise the FIFO queue (which now only engages once the pool is full).
test_cancel_event_checkandtest/test_mcp_core_audit_e.pyare updated to use the ContextVar API.Manual verification
N/A — unit coverage is sufficient: the convoy regression, the per-worker cancel isolation, the two new refusals and the drain-vs-cancel distinction are all pinned by tests, and each production fix has a mutation probe confirming its test fails without it.
Related Issues
Fixes #17031
Pattern harvest
Rule candidate: review-prompt
Pattern: a module-global "current in-flight" slot (cancel flag, caller, request id) that is safe only because dispatch is single-worker — introducing concurrency silently corrupts it unless the slot is first moved to a
ContextVaror a per-request record. A second instance of the same class: a drain/exit signal that reuses a cooperative tool's existing cancel-check path (is_tool_cancelled()) must be distinguishable from a REAL cancel at the point the exception is handled, or "the process is shutting down" and "the caller asked to stop" collapse into the same (wrong, for the first case) outcome.Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)