Repository navigation
fix(mcp): park waiting tools off the worker so a wait stalls nobody - #18841
Conversation
UX Review (Opus 5.5) — ⏭️ skippedRevision |
First Principles Review (Opus 5.5) — 🟡 CONCERNSPremise-level review of First-Principles-Verdict: CONCERNS Deferring Not justified as shipped
What this change shipsInventory (9 items) — 7 justifiedIntent: stop one session's long tool wait from stalling every other session on the shared core backend, and recycle a backend that really is stuck within minutes. FIX (#17031).
[FIRST-PRINCIPLES-REVIEWED] 9448e96 Description read[DESCRIPTION-READ] 9c42f15ce38df2c181fe76cdfa35e75e17ac8d2857d7854375f5b769a49eaf5a That sha256 names the PR description this verdict read. Recompute it with |
Design Review (Opus 5.5) — 🟡 CONCERNSDesign-level review of Design-Verdict: CONCERNS Nothing enforces the 900s ceiling's premise that no synchronous Watch
Suggestions
[DESIGN-REVIEWED] 9448e96 Description read[DESCRIPTION-READ] 85deed07bf27c466cd02788b33bd50789f5613cf73d2f39886c755d188d318ac That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
ff5ae88 to
0408ccd
Compare
0408ccd to
57c05d2
Compare
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
2c3060b to
0b9011b
Compare
0b9011b to
c9c9a2f
Compare
|
|
|
c9c9a2f to
abd2634
Compare
|
|
|
|
Rebased onto origin/main 2a0400d (same-file overlap: .github/black-baseline.txt; one content conflict in src/kiro_crew/mcp_shared.py from #18979, resolved by porting its change -- any id-bearing request while the worker is busy is answered through _dispatch -- into this PR's busy branch; #18979's test passes). No other change since 18452da, on which GPT and Opus passed. |
|
|
|
|
|
|
|
|
|
|
Rebased onto origin/main 22a7358 (+70 commits). Two content conflicts resolved: |
|
|
buluoray
left a comment
There was a problem hiding this comment.
Requesting changes at eb2eab4 for one doc line only. Auto-merge is on, so this review is the last gate before main.
Blocking
src/kiro_crew/docs/agent-questions.md:52(shipped user doc) still draws the wait loop as "20 s slices". This PR changesASK_WAIT_SLICE_SECSfrom 20 to 5 and updates the system spec to match, so the user-facing doc now contradicts it. Please change it to 5 s in this PR.
While you are in there, follow-up 1 below is worth adding in the same push: it is the only guard on the cross-session path with no test.
Everything else holds up against the checks that matter for a shared backend:
- Parked state is per caller. The parked table and step pool are locals of the dispatch loop; everything the PR adds at module level is a constant. Entries are keyed by the gateway's unique forward id, so a settled result goes back to the stub that made the call. Every step, cancel hook, abandon hook and settle commit runs with that call's own caller and tenant nonce, and clears them afterwards.
- Every parked wait is bounded.
- The parked table caps at
DEFERRED_PARKED_MAX, with a retryable -32000 on overflow. notifications/cancelledwithdraws a queued step, or marks a running one and discards its result, then runscancel()with no response.- A pruned-install exit lets an in-flight step finish within
DEFERRED_STEP_STUCK_SECSbefore answering or refusing. - The loop's
finallycancels queued futures and shuts the pool down. - On the gateway, a reporting backend recycles at
PROGRESS_WEDGE_CEILING_SECS, and a cancelled entry is skipped only by the hard ceiling, so the two-condition ping rule still applies to it.
- The parked table caps at
- The
waitearly end is preserved. A keepalive goes out everyWAIT_PING_SECS, a reply must name thiswait_idto end the sleep, and the early end is a normal result rather thanToolCancelled. - Upgrade-safe. The report is advertised only by an opted-in server on POSIX (
platform_compat.IS_POSIX), and onlykirocrew-coreopts in. The gateway lowers the ceiling only for a backend that advertised it._call_tooland the Windows path still drive a deferred tool inline.
Guards checked by mutation: removing the cancel hook, the parked-table bound, the short ceiling, the cancelled-entry skip, the wait_id match or the mid-step cancel drop each turns a named test red.
Follow-ups (non-blocking except where noted above):
- No test pins that a deferred step runs under its own call's caller identity. Replacing
set_current_caller(entry.caller_ctx)in_run_stepwithNoneleaves every touched suite green. A loop-level test should assert the identity a step sees, astest_spawn_collection_commit_installs_its_call_identitydoes for the settle commit. This is the guard that keeps a parkedwait's keepalive on its own session in a pooled backend. - Cancel hooks make bounded HTTP calls (5s each) on the dispatch thread. If a stub with many parked calls disconnects while the gateway is unresponsive, ping answers can be held for up to
DEFERRED_PARKED_MAX × 5s. Consider moving the hooks onto the step pool, or capping the total hook time per tick. - On EOF, parked calls'
cancel()hooks are not run, so an open question card is left to the stale reaper. This matches the behaviour before this change, but it is worth a comment or a best-effort pass.
|
Addressed at 4f3e422 (rebased onto Blocking
Follow-up 1 (taken, same push)
Follow-ups 2 and 3: noted, not taken here. Both are pre-existing shapes (cancel hooks already ran on the dispatch thread for a worker-path cancel; EOF already left cards to the stale reaper) and sit outside the frozen goal. Happy to take them as a follow-up PR if you want them. Also in this push: the ratchet ( |
The pooled kirocrew-core backend runs tool calls on one worker thread, and wait, spawn_sub_agents and ask_question spent their whole duration asleep on it. One session parked on a 1800s wait queued every other session's call behind it, and because the loop still answered pings the gateway's wedge detector held the backend healthy for the 3h ceiling. A tool that only waits now returns a DeferredTool: setup runs on the worker, the waiting is short steps the dispatch loop times from its own tick and runs on a small pool, so the worker is free the moment the handler returns and a thousand parked sessions hold no thread. The loop reports its parked ids to a gateway that asked for them in initialize; the gateway ages those requests from the last report and holds a reporting backend to a 900s ceiling instead of 3h, so a genuinely hung call recycles in minutes. Closes #17031
|
|
|
|
buluoray
left a comment
There was a problem hiding this comment.
Approve at 9448e96985. This supersedes my earlier request changes at eb2eab454b.
What this brings users: one session's wait or ask_question no longer stalls other sessions' tools on the shared MCP worker, and a stuck backend is recycled in about 15 minutes instead of 3 hours.
Both asks from my last review are done
- Doc:
src/kiro_crew/docs/agent-questions.md:52now says5 s slices, matching the new slice. - Identity test:
TestStepRunsUnderItsOwnIdentityparks two calls from two sessions and checks every step sees its own caller and tenant nonce, and that the pool thread is cleared between steps.- Mutation: making
_run_step(src/kiro_crew/mcp_shared.py:2725-2738) install no caller turnstest_each_step_sees_its_call_identity_and_the_pool_thread_is_clearedred. Restored, the file passes (49 tests).
- Mutation: making
Range-diff
Besides the two items above, the only change is in test_mcp_gateway_wedge_ping_gate.py: the 0.05 s wait is now documented as a tick pace with the tick count as the real bound. No product code changed.
CI on this head: 89 success, 9 skipped, 0 failure; legacy statuses (Linux, Windows, PR Readiness) all success.
Problem / Motivation
Goal: One session's
wait,spawn_sub_agentsorask_questionnever makes another session'skirocrew-coretool call queue behind it, and a backend that really is stuck is recycled in minutes, not hours.The pooled
kirocrew-corebackend serves every dashboard session from one process with one tool worker thread. A session parked onwait(up to 1800s) orspawn_sub_agents(up to 7200s) slept on that thread the whole time, so everyone else'smemory_recall,session_ledger_recordandresource_statuswaited behind it -- for minutes to hours (#17031). The loop kept answering the gateway'sping, so the wedge detector saw a healthy backend and did nothing; the only exit was the 3-hour hard ceiling or an operator killing the pid.Why it matters
Every dashboard session shares that backend. One babysit loop's
waitstalls every other chat's tools, new sessions time out onsession/newand come back untrusted, and the operator's only remedy is a manual kill. A worker pool would only move the threshold from one parked call to N.Not a goal
kirocrew-coreprocess under load.session/newso it does not wait on the pooled backend's queue.What changed (motivation → approach → change)
Before,
wait,spawn_sub_agentsorask_questionheld the pooled backend's only worker thread for the whole wait, so every other session'skirocrew-corecall queued behind it, and the gateway never recycled the backend because it still answered pings. Now the wait holds no thread, and a stuck backend is recycled in 15 minutes.waitDeferredTool; one keepalive POST per step, bounded to 5s (_WAIT_PING_TIMEOUT_SECS)spawn_sub_agentsask_questionASK_WAIT_SLICE_SECS20→5 so 64 cards over 8 step threads queue a poll by ≤40s, under the 120s gracespawn_runkirocrew-core)A thread per waiting call (capped at 64) was rejected with the pool: it still blocks a thread per parked wait, cannot be cancelled except cooperatively, and recycles the backend at the same 64-call cliff. A handler that only waits returns a
mcp_shared.DeferredTool; the dispatch loop runs its shortstep()from its 100 ms tick wheneverdue_at()has passed; the step that returns text answers the call. Steps run on a fixed pre-started pool (_StepPool, 8 threads) under the call's identity; a stuck step never delays the loop. Anotifications/cancelledfor a parked call withdraws a queued step, drops the call unanswered and runs itscancel()hook. The parked table is bounded byDEFERRED_PARKED_MAX(64); an overflow is cancelled and answered with a retryable-32000. A parked step retains its member list until submission finishes, soSPAWN_BATCH_MEMBERS_MAX(128) boundstasks/agents, refused before any deferred state exists; error lines are bounded (_SA_ERROR_MAX_CHARS); only two clamped string fields per member are kept; a non-stringpromptoragent_or_modeis refused withError: ... must be a string. Aspawn_runcancelled mid-submission reconciles its unsubmitted tasks (_close_unsubmitted). What must happen once (children marked collected, the tool's own SEL row) runs inon_settled, only when the result is accepted. A call that will never step again (table full, pool cannot start, pruned exit) is first askedabandon(): a batch whose children already run answers naming them; one the client already cancelled gets no response, only acancelledrow._call_tooland Windows drive a deferred inline.The in-flight report closes the self-heal gap. A loop started with
reports_inflight=True-- onlyrun_mcp_core_serverpasses it; on a synchronous-only server it would only lower the ceiling under the running call -- advertiseskirocrew.inflight-reportininitializeand, when the client declared it too, sendsnotifications/kirocrew/inflightevery 30s naming the parked ids still progressing and those queued behind the worker. The running call is not named: its stamp freezes at its start, so the ceiling bounds one call's run; once cancelled, the queued calls age instead. The gateway asks for it on every forwardedinitialize, stamps_PendingRequest.t_progress_ms, andBackend._heartbeat_onceholds a reporting backend toPROGRESS_WEDGE_CEILING_SECS(900s). Forwarding a stub'snotifications/cancelledmarks the pending entry cancelled (resource subscribe/unsubscribe excepted): the backend answers it with no response, so the hard ceiling skips it while the two-condition ping rule, attribution and the last-stub recycle still see it as on main.Where:
mcp_shared.py(DeferredTool,drive_deferred,_StepPool, report sender);mcp_tools/control.py(_WaitStep,_AskStep);mcp_tools/spawn.py(_SubAgentsStep,_SpawnRunStep);validation.py;mcp_core.py;mcp_caller.py;mcp_gateway/backend.py;test_mcp_core_audit_e.pyleaves.github/black-baseline.txt; docsmcp.md,subagent.md,learn-cron-dashboard.md.Backwards compatibility
Compatible: every tool's arguments and result text are unchanged;
_call_toolstill returns a string;/api/agent-ask/{id}/waitstill reads no body; the report is sent only by a loop that opted in AND to a client that declared it, and the short ceiling applies only to a backend that advertised it, so an older gateway or backend behaves as before; every other server keeps the long ceiling.Refuses what was accepted before:
spawn_sub_agents.agents,spawn_run.tasksorspawn_run.agentsoverSPAWN_BATCH_MEMBERS_MAX(128); readers: the schemas inmcp_tools/spawn.py; no stored record changes.ASK_WAIT_SLICE_SECS20→5; readers:dashboard/handlers/ask_question.py:api_agent_ask_wait,_AskStep; a caller still cannot choose it (test_wait_ignores_the_request_body).Removes
mcp_shared.is_tool_cancelledand_thread_cancel_event: their three callers are now deferred and nothing insrc/calls them; a step may still raiseToolCancelled. Each tool still writes its own SEL row (waitsuccess,spawn_sub_agentscompleted/partial) beside the wrapper's generic row. No path, record, key or wire shape goes away. Readers, each now aDeferredToolwith acancel()hook:_WaitStep;test_mcp_core_more_coverage.py::TestWaitTool::test_cancellation_retires_the_card_and_never_reports_success_AskStep;test_ask_question_blocking.py_SubAgentsStep;test_mcp_core_audit_e.py::TestSpawnSubAgentsCancellation::test_poll_loop_honors_cancellationToolCancelled;test_mcp_gateway_wedge_ping_gate.py::test_tool_cancelled_exception_suppresses_responseTests
test/test_mcp_shared_deferred_tools.py(new): a parked call does not block the next call; nine parked calls hold no worker and all settle; a step blocked on the gateway leaves pings and other calls served; cancel drops a parked call with no response and audits once; a cancel mid-step discards that step's result; a step raisingToolCancelledends silently; a step raising answers an error and the loop continues; the in-flight report names parked ids only to a client that asked, omits a call stuck inside a step, and names a call queued behind the worker until it starts running, but not once the running call has been cancelled; an overflow pastDEFERRED_PARKED_MAXis cancelled, audited and refused while admitted calls still settle and capacity frees for a retry; a cancel hook (idle, mid-step, after a step error, late-worker, pruned-exit) observes the call's own caller and tenant nonce and leaves neither installed, even when the hook raises; every step runs with its OWN call's caller and tenant nonce and the pool thread is cleared between steps (TestStepRunsUnderItsOwnIdentity; replacing the install in_run_stepwithNonereds it);on_settledruns with the call's identity and never after a mid-step cancel; on a pruned exit, on table overflow and when the step pool cannot start, a call whoseabandon()returns text gets it as a normal result (oneon_settled, no failed or rejected row, no cancel hook) whileNoneor a raising hook keeps the retryable error, and the parked table never exceeds its bound in either case; a step the pool cannot start fails only its own call; a pruned-install exit finishes and delivers a step in flight and refuses only the idle parked call; the SEL row is committed by the driver at settle, never by a step, anddrive_deferredcommits it inline;drive_deferredsleeps to each due time; the Windows path collapses a deferred inline;map_deferredmaps the settling text only.test/test_mcp_core_spawn_sub_agents.py: the handler returns the deferred before any/api/spawnPOST and each submit step posts one member, a simulated 128-member batch at 8s each staysprogressing, the two early error texts come from the step and mark nothing collected, andabandon()mid-submit names the accepted ids and the count not yet submitted, returnsNone(so the call stays retryable) only while no child is accepted and no member's POST failed in transport, and otherwise lists members of unknown acceptance so they are not blindly resubmitted; a collect step has no commit side effect; settle marks children collected once with the caller's identity;abandon()names every child and marks nothing; a member's extra keys are dropped, not retained or refused, and its strings clamped; member prompts are dropped once the submit phase ends; 128 members accepted and 129 refused before any spawn (test_batch_member_limit_is_checked_before_spawning); error lines truncated; the tool's ownpartialrow withspawned/completed/still_running/erroredis written once at settle beside the genericcompletedrow (test_finished_child_that_ended_in_error_is_still_counted_as_errored, as on main).test/test_ask_question_blocking.py: the pinnedtest_wait_ignores_the_request_bodyandtest_the_wait_carries_no_slice_the_server_would_honourare kept (every wait POST carries{}; the slice is the server's constant whatever the body);test_mcp_shared_deferred_tools.py::test_ask_queue_budget_is_inside_the_outcome_retention_gracepinsceil(DEFERRED_PARKED_MAX / DEFERRED_STEP_WORKERS) * ASK_WAIT_SLICE_SECSunder_AGENT_ASK_STALE_SECS.test/test_subagent_context_group_boundary.py: drives the deferred's submit phase before reading the posted bodies.spawn_run: the handler returns the deferred before any POST, a one-task call answers after one step, a simulated 128-task batch at 8s each staysprogressing, stored lines truncated,abandon()keeps the unknown-acceptance warning when a submission timed out, a refused agent name still skips its siblings and reconciles them lost, 128 tasks or agents accepted and 129 refused before any POST,cancel()/abandon()mid-submission reconcile each unsubmitted task once and an unanswering gateway costs one bounded report (test_spawn_run_per_task_overrides.py),abandon()keeps accepted and queued ids, and the receipt is byte-identical to the old loop's (test_spawn_run_per_task_overrides.py,test_spawn_agent_roster.py,test_spawn_reasoning_effort.py,test_spawn_queued_reason.py,test_spawn_single_task_gate.py,test_subagent_no_tool_calls_note.py,test_spawn_backend.pydrive the deferred instead of reading the string directly).test/test_mcp_gateway_inflight_report.py(new):test_ceiling_outlives_every_synchronous_tool_boundscans everytimeout=and*_TIMEOUT_SECSundermcp_tools/andmcp_core.pyby AST and fails if one reaches the 900s ceiling; the forwardedinitializeasks for the report without aliasing the stub's frame; the report refreshest_progress_msand is neither routed to a stub nor recorded as a hazard; a malformed report refreshes nothing; a reported request is aged from its report; a reporting backend recycles at 900s and a legacy backend keeps the 3h ceiling; several slow synchronous calls queued behind a reporting backend, each within its own bound, are not recycled while a single call running past the ceiling is; an unreported request on a small monotonic clock is still aged from its start; a forwardednotifications/cancelledmarks the entry and keeps it (another stub's cancel does not), a cancelled entry older than the short ceiling leaves the heartbeatalivewhile an uncancelled one of the same age is wedged, yet with pings stale it still wedges on every backend kind, a late frame for it routes to its sole holder or, on a shared token, to nobody with a hazard as on main, a late response settles it, and a cancelled call still makes the sole stub recyclable (TestCancelledCallMarking,test_forwarded_cancel_then_disconnect_still_recycles); one/api/spawnPOST's timeout is underDEFERRED_STEP_STUCK_SECSand the short ceiling; a stale report does not outlive the short ceiling; the three ceilings are ordered and the short one exceeds every synchronous tool bound found by an AST scan oftimeout=arguments and*_TIMEOUT_SECSconstants undermcp_tools/andmcp_core.py(23 bounds, max_POD_UP_TIMEOUT_S); other unattributable notifications still record a hazard.test_mcp_gateway_wedge_ping_gate.py(the cancel-suppression test no longer depends on the deleted function and waits for a later call to be answered before asserting; its 50 ms tick pace carries aflake-ok, since expiry only yields another tick),test_mcp_shared.py(a loop advertises the report only when started withreports_inflight=True; the default advertises nothing),test_mcp_shared_deferred_tools.py::test_a_server_that_did_not_opt_in_neither_advertises_nor_reports,test_mcp_core_more_coverage.py::test_only_kirocrew_core_opts_into_the_inflight_report(coreTrue; cron, work, guide, panelFalse),test_mcp_core_spawn_sub_agents.pyandtest_wait_tool_early_end.py(fake clocks no longer pin a call count),test_fairness_lanes.py,test_spawn_queued_reason.py,test_spawn_single_task_gate.py(drive the deferred throughdrive_deferredinstead of patchingis_tool_cancelled),test_ask_question_blocking.py,test_mcp_core_more_coverage.py,test_mcp_core_audit_e.py(cancellation reaches the deferred'scancel()hook instead of a patchedis_tool_cancelled).Manual verification
N/A — unit coverage sufficient: the loop-level tests drive the real
run_mcp_stdio_loopover a pipe with a parked call and a concurrent one, and the gateway tests drive the realBackend._heartbeat_onceand_route_backend_line.Screenshots / video
N/A — no user-visible UI change.
Pattern harvest
Rule candidate: review-prompt
Pattern: a tool handler that only waits (sleep, poll, long-poll) runs on the MCP server's single worker thread; on a pooled backend that is a convoy for every other caller.
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)Related Issues
Closes #17031