Skip to content

fix(mcp): park waiting tools off the worker so a wait stalls nobody - #18841

Merged
bolichen97 merged 1 commit into
mainfrom
fix/mcp-deferred-long-tools
Oct 12, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
fix/mcp-deferred-long-tools

Conversation

@bolichen97

@bolichen97 bolichen97 commented Oct 10, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: One session's wait, spawn_sub_agents or ask_question never makes another session's kirocrew-core tool call queue behind it, and a backend that really is stuck is recycled in minutes, not hours.

The pooled kirocrew-core backend serves every dashboard session from one process with one tool worker thread. A session parked on wait (up to 1800s) or spawn_sub_agents (up to 7200s) slept on that thread the whole time, so everyone else's memory_recall, session_ledger_record and resource_status waited behind it -- for minutes to hours (#17031). The loop kept answering the gateway's ping, so the wedge detector saw a healthy backend and did nothing; the only exit was the 3-hour hard ceiling or an operator killing the pid.

Why it matters

Every dashboard session shares that backend. One babysit loop's wait stalls every other chat's tools, new sessions time out on session/new and come back untrusted, and the operator's only remedy is a manual kill. A worker pool would only move the threshold from one parked call to N.

Not a goal

What changed (motivation → approach → change)

Before, wait, spawn_sub_agents or ask_question held the pooled backend's only worker thread for the whole wait, so every other session's kirocrew-core call queued behind it, and the gateway never recycled the backend because it still answered pings. Now the wait holds no thread, and a stuck backend is recycled in 15 minutes.

Part Before After
wait sleeps 60-1800s on the worker DeferredTool; one keepalive POST per step, bounded to 5s (_WAIT_PING_TIMEOUT_SECS)
spawn_sub_agents submits, then polls children every 2s, on the worker deferred: one POST per step, then one poll / probe / collect per step
ask_question long-polls the server's fixed 20s slice on the worker deferred; one slice per step; ASK_WAIT_SLICE_SECS 20→5 so 64 cards over 8 step threads queue a poll by ≤40s, under the 120s grace
spawn_run submits every task on the worker deferred: one POST per step, the same receipt after the last
the worker held for the wait free once the handler returns
gateway ceiling 3h05m for every backend 900s for a reporting backend (only kirocrew-core)

A thread per waiting call (capped at 64) was rejected with the pool: it still blocks a thread per parked wait, cannot be cancelled except cooperatively, and recycles the backend at the same 64-call cliff. A handler that only waits returns a mcp_shared.DeferredTool; the dispatch loop runs its short step() from its 100 ms tick whenever due_at() has passed; the step that returns text answers the call. Steps run on a fixed pre-started pool (_StepPool, 8 threads) under the call's identity; a stuck step never delays the loop. A notifications/cancelled for a parked call withdraws a queued step, drops the call unanswered and runs its cancel() hook. The parked table is bounded by DEFERRED_PARKED_MAX (64); an overflow is cancelled and answered with a retryable -32000. A parked step retains its member list until submission finishes, so SPAWN_BATCH_MEMBERS_MAX (128) bounds tasks/agents, refused before any deferred state exists; error lines are bounded (_SA_ERROR_MAX_CHARS); only two clamped string fields per member are kept; a non-string prompt or agent_or_mode is refused with Error: ... must be a string. A spawn_run cancelled mid-submission reconciles its unsubmitted tasks (_close_unsubmitted). What must happen once (children marked collected, the tool's own SEL row) runs in on_settled, only when the result is accepted. A call that will never step again (table full, pool cannot start, pruned exit) is first asked abandon(): a batch whose children already run answers naming them; one the client already cancelled gets no response, only a cancelled row. _call_tool and Windows drive a deferred inline.

The in-flight report closes the self-heal gap. A loop started with reports_inflight=True -- only run_mcp_core_server passes it; on a synchronous-only server it would only lower the ceiling under the running call -- advertises kirocrew.inflight-report in initialize and, when the client declared it too, sends notifications/kirocrew/inflight every 30s naming the parked ids still progressing and those queued behind the worker. The running call is not named: its stamp freezes at its start, so the ceiling bounds one call's run; once cancelled, the queued calls age instead. The gateway asks for it on every forwarded initialize, stamps _PendingRequest.t_progress_ms, and Backend._heartbeat_once holds a reporting backend to PROGRESS_WEDGE_CEILING_SECS (900s). Forwarding a stub's notifications/cancelled marks the pending entry cancelled (resource subscribe/unsubscribe excepted): the backend answers it with no response, so the hard ceiling skips it while the two-condition ping rule, attribution and the last-stub recycle still see it as on main.

Where: mcp_shared.py (DeferredTool, drive_deferred, _StepPool, report sender); mcp_tools/control.py (_WaitStep, _AskStep); mcp_tools/spawn.py (_SubAgentsStep, _SpawnRunStep); validation.py; mcp_core.py; mcp_caller.py; mcp_gateway/backend.py; test_mcp_core_audit_e.py leaves .github/black-baseline.txt; docs mcp.md, subagent.md, learn-cron-dashboard.md.

Backwards compatibility

Compatible: every tool's arguments and result text are unchanged; _call_tool still returns a string; /api/agent-ask/{id}/wait still reads no body; the report is sent only by a loop that opted in AND to a client that declared it, and the short ceiling applies only to a backend that advertised it, so an older gateway or backend behaves as before; every other server keeps the long ceiling.

Refuses what was accepted before: spawn_sub_agents.agents, spawn_run.tasks or spawn_run.agents over SPAWN_BATCH_MEMBERS_MAX (128); readers: the schemas in mcp_tools/spawn.py; no stored record changes.

ASK_WAIT_SLICE_SECS 20→5; readers: dashboard/handlers/ask_question.py:api_agent_ask_wait, _AskStep; a caller still cannot choose it (test_wait_ignores_the_request_body).

Removes mcp_shared.is_tool_cancelled and _thread_cancel_event: their three callers are now deferred and nothing in src/ calls them; a step may still raise ToolCancelled. Each tool still writes its own SEL row (wait success, spawn_sub_agents completed/partial) beside the wrapper's generic row. No path, record, key or wire shape goes away. Readers, each now a DeferredTool with a cancel() hook:

  • Reader: src/kiro_crew/mcp_tools/control.py:wait -- chat -- _WaitStep; test_mcp_core_more_coverage.py::TestWaitTool::test_cancellation_retires_the_card_and_never_reports_success
  • Reader: src/kiro_crew/mcp_tools/control.py:_await_ask_outcome -- chat -- _AskStep; test_ask_question_blocking.py
  • Reader: src/kiro_crew/mcp_tools/spawn.py:spawn_sub_agents -- subagent -- _SubAgentsStep; test_mcp_core_audit_e.py::TestSpawnSubAgentsCancellation::test_poll_loop_honors_cancellation
  • Reader: src/kiro_crew/mcp_shared.py:_run_worker -- chat -- still catches ToolCancelled; test_mcp_gateway_wedge_ping_gate.py::test_tool_cancelled_exception_suppresses_response

Tests

  • test/test_mcp_shared_deferred_tools.py (new): a parked call does not block the next call; nine parked calls hold no worker and all settle; a step blocked on the gateway leaves pings and other calls served; cancel drops a parked call with no response and audits once; a cancel mid-step discards that step's result; a step raising ToolCancelled ends silently; a step raising answers an error and the loop continues; the in-flight report names parked ids only to a client that asked, omits a call stuck inside a step, and names a call queued behind the worker until it starts running, but not once the running call has been cancelled; an overflow past DEFERRED_PARKED_MAX is cancelled, audited and refused while admitted calls still settle and capacity frees for a retry; a cancel hook (idle, mid-step, after a step error, late-worker, pruned-exit) observes the call's own caller and tenant nonce and leaves neither installed, even when the hook raises; every step runs with its OWN call's caller and tenant nonce and the pool thread is cleared between steps (TestStepRunsUnderItsOwnIdentity; replacing the install in _run_step with None reds it); on_settled runs with the call's identity and never after a mid-step cancel; on a pruned exit, on table overflow and when the step pool cannot start, a call whose abandon() returns text gets it as a normal result (one on_settled, no failed or rejected row, no cancel hook) while None or a raising hook keeps the retryable error, and the parked table never exceeds its bound in either case; a step the pool cannot start fails only its own call; a pruned-install exit finishes and delivers a step in flight and refuses only the idle parked call; the SEL row is committed by the driver at settle, never by a step, and drive_deferred commits it inline; drive_deferred sleeps to each due time; the Windows path collapses a deferred inline; map_deferred maps the settling text only.
  • test/test_mcp_core_spawn_sub_agents.py: the handler returns the deferred before any /api/spawn POST and each submit step posts one member, a simulated 128-member batch at 8s each stays progressing, the two early error texts come from the step and mark nothing collected, and abandon() mid-submit names the accepted ids and the count not yet submitted, returns None (so the call stays retryable) only while no child is accepted and no member's POST failed in transport, and otherwise lists members of unknown acceptance so they are not blindly resubmitted; a collect step has no commit side effect; settle marks children collected once with the caller's identity; abandon() names every child and marks nothing; a member's extra keys are dropped, not retained or refused, and its strings clamped; member prompts are dropped once the submit phase ends; 128 members accepted and 129 refused before any spawn (test_batch_member_limit_is_checked_before_spawning); error lines truncated; the tool's own partial row with spawned/completed/still_running/errored is written once at settle beside the generic completed row (test_finished_child_that_ended_in_error_is_still_counted_as_errored, as on main). test/test_ask_question_blocking.py: the pinned test_wait_ignores_the_request_body and test_the_wait_carries_no_slice_the_server_would_honour are kept (every wait POST carries {}; the slice is the server's constant whatever the body); test_mcp_shared_deferred_tools.py::test_ask_queue_budget_is_inside_the_outcome_retention_grace pins ceil(DEFERRED_PARKED_MAX / DEFERRED_STEP_WORKERS) * ASK_WAIT_SLICE_SECS under _AGENT_ASK_STALE_SECS. test/test_subagent_context_group_boundary.py: drives the deferred's submit phase before reading the posted bodies. spawn_run: the handler returns the deferred before any POST, a one-task call answers after one step, a simulated 128-task batch at 8s each stays progressing, stored lines truncated, abandon() keeps the unknown-acceptance warning when a submission timed out, a refused agent name still skips its siblings and reconciles them lost, 128 tasks or agents accepted and 129 refused before any POST, cancel()/abandon() mid-submission reconcile each unsubmitted task once and an unanswering gateway costs one bounded report (test_spawn_run_per_task_overrides.py), abandon() keeps accepted and queued ids, and the receipt is byte-identical to the old loop's (test_spawn_run_per_task_overrides.py, test_spawn_agent_roster.py, test_spawn_reasoning_effort.py, test_spawn_queued_reason.py, test_spawn_single_task_gate.py, test_subagent_no_tool_calls_note.py, test_spawn_backend.py drive the deferred instead of reading the string directly).
  • test/test_mcp_gateway_inflight_report.py (new): test_ceiling_outlives_every_synchronous_tool_bound scans every timeout= and *_TIMEOUT_SECS under mcp_tools/ and mcp_core.py by AST and fails if one reaches the 900s ceiling; the forwarded initialize asks for the report without aliasing the stub's frame; the report refreshes t_progress_ms and is neither routed to a stub nor recorded as a hazard; a malformed report refreshes nothing; a reported request is aged from its report; a reporting backend recycles at 900s and a legacy backend keeps the 3h ceiling; several slow synchronous calls queued behind a reporting backend, each within its own bound, are not recycled while a single call running past the ceiling is; an unreported request on a small monotonic clock is still aged from its start; a forwarded notifications/cancelled marks the entry and keeps it (another stub's cancel does not), a cancelled entry older than the short ceiling leaves the heartbeat alive while an uncancelled one of the same age is wedged, yet with pings stale it still wedges on every backend kind, a late frame for it routes to its sole holder or, on a shared token, to nobody with a hazard as on main, a late response settles it, and a cancelled call still makes the sole stub recyclable (TestCancelledCallMarking, test_forwarded_cancel_then_disconnect_still_recycles); one /api/spawn POST's timeout is under DEFERRED_STEP_STUCK_SECS and the short ceiling; a stale report does not outlive the short ceiling; the three ceilings are ordered and the short one exceeds every synchronous tool bound found by an AST scan of timeout= arguments and *_TIMEOUT_SECS constants under mcp_tools/ and mcp_core.py (23 bounds, max _POD_UP_TIMEOUT_S); other unattributable notifications still record a hazard.
  • Updated: test_mcp_gateway_wedge_ping_gate.py (the cancel-suppression test no longer depends on the deleted function and waits for a later call to be answered before asserting; its 50 ms tick pace carries a flake-ok, since expiry only yields another tick), test_mcp_shared.py (a loop advertises the report only when started with reports_inflight=True; the default advertises nothing), test_mcp_shared_deferred_tools.py::test_a_server_that_did_not_opt_in_neither_advertises_nor_reports, test_mcp_core_more_coverage.py::test_only_kirocrew_core_opts_into_the_inflight_report (core True; cron, work, guide, panel False), test_mcp_core_spawn_sub_agents.py and test_wait_tool_early_end.py (fake clocks no longer pin a call count), test_fairness_lanes.py, test_spawn_queued_reason.py, test_spawn_single_task_gate.py (drive the deferred through drive_deferred instead of patching is_tool_cancelled), test_ask_question_blocking.py, test_mcp_core_more_coverage.py, test_mcp_core_audit_e.py (cancellation reaches the deferred's cancel() hook instead of a patched is_tool_cancelled).

Manual verification

N/A — unit coverage sufficient: the loop-level tests drive the real run_mcp_stdio_loop over a pipe with a parked call and a concurrent one, and the gateway tests drive the real Backend._heartbeat_once and _route_backend_line.

Screenshots / video

N/A — no user-visible UI change.

Pattern harvest

Rule candidate: review-prompt
Pattern: a tool handler that only waits (sleep, poll, long-poll) runs on the MCP server's single worker thread; on a pooled backend that is a convoy for every other caller.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Related Issues

Closes #17031

@bolichen97
bolichen97 requested review from a team and KiroCrewCaptain as code owners October 10, 2026 11:46
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 10, 2026
@github-actions

github-actions Bot commented Oct 10, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 9448e96985e1b5b9fe023489c59ff746520a6ede touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Advisory — does not block merge.

@github-actions

github-actions Bot commented Oct 10, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — 🟡 CONCERNS

Premise-level review of 9448e96985e1b5b9fe023489c59ff746520a6ede — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: CONCERNS

Deferring spawn_run is outside the frozen Goal. That Goal names wait, spawn_sub_agents and ask_question, and spawn_run waits on nothing, yet it brings its own reconcile and abandon surface.

Not justified as shipped

  • Item 4 — oversized: the Goal is that "wait, spawn_sub_agents or ask_question never makes another session's … call queue behind it". spawn_run only "submits every task on the worker". It never parks on a wait, and ## Not a goal keeps "the one-worker rule" for every other call. Converting it still adds _SpawnRunStep, _close_unsubmitted, an abandon() receipt, _LOST_REPORT_TIMEOUT_SECS, and a _reconcile_lost that now returns bool.
    Subtraction: delete _SpawnRunStep and the bounded lost-report change, and keep spawn_run synchronous as it is on main.
    Clears when: the spawn_run deferral is removed, or a human amends the Goal to cover spawn_run.
  • Item 8 — rides along: the new Error: ... must be a string refusal for a non-string prompt/agent_or_mode is not part of the fix.

What this change ships

Inventory (9 items) — 7 justified

Intent: stop one session's long tool wait from stalling every other session on the shared core backend, and recycle a backend that really is stuck within minutes. FIX (#17031).

  1. wait no longer holds the shared worker while it waits — justified
  2. spawn_sub_agents no longer holds the worker while it polls its children — justified
  3. ask_question no longer holds the worker; the answer poll drops from 20s to 5s — justified
  4. spawn_run posts its tasks in background steps instead of on the worker — oversized
  5. A stuck core backend is recycled after 900s instead of 3h — justified
  6. A cancelled call no longer ages the backend into the recycle ceiling — justified
  7. More than 64 parked calls are refused with a retryable error — justified
  8. Non-string member prompt/agent_or_mode refused with an explicit error — rides along
  9. Spawn batches over 128 members are refused — justified

[FIRST-PRINCIPLES-REVIEWED] 9448e96

Description read

[DESCRIPTION-READ] 9c42f15ce38df2c181fe76cdfa35e75e17ac8d2857d7854375f5b769a49eaf5a

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 10, 2026
@github-actions

github-actions Bot commented Oct 10, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — 🟡 CONCERNS

Design-level review of 9448e96985e1b5b9fe023489c59ff746520a6ede — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: CONCERNS

Nothing enforces the 900s ceiling's premise that no synchronous kirocrew-core tool runs that long, so a slow new tool would recycle the backend every dashboard session shares.

Watch

  • The short ceiling rests on an unwritten rule. The gateway comment on PROGRESS_WEDGE_CEILING_SECS says "no shipped tool does [run a quarter-hour] on purpose (the longest synchronous one bounds itself at 210s)". Only a comment holds that rule; no code checks it.
    • Cause: the running worker call is deliberately left out of the report, so it ages from its start.
    • Mechanism: a reporting backend now uses PROGRESS_WEDGE_CEILING_SECS if self.reports_inflight. Any future core tool with a synchronous path longer than 900s (an indexer, an app pod bring-up whose budget gets raised, a timeout= someone raises) trips that hard ceiling regardless of ping freshness.
    • Consequence: one slow but healthy call broadcasts backend gone to every pooled session and drops every parked wait and spawn_sub_agents with it. That is a wider outage than the stall this PR removes.
    • The 3h ceiling avoided this by deriving from SUBAGENT_TIMEOUT_SECS. The new value has no such source.
    • Clears when: a test pins every non-deferred kirocrew-core tool's own timeout budget below PROGRESS_WEDGE_CEILING_SECS (or the ceiling is derived from the largest such budget), so a new long synchronous tool fails CI instead of recycling production backends.

Suggestions

  • Write the budget down as one data point, a SYNC_TOOL_MAX_SECS constant next to DEFERRED_STEP_STUCK_SECS. Derive the 900s ceiling from it, and have tools with long bounds (_POD_UP_TIMEOUT_S, _ADD_DOCUMENT_TIMEOUT_SECS) assert they stay under it.

[DESIGN-REVIEWED] 9448e96

Description read

[DESCRIPTION-READ] 85deed07bf27c466cd02788b33bd50789f5613cf73d2f39886c755d188d318ac

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 10, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 9448e96985e1b5b9fe023489c59ff746520a6ede and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 9448e96

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 9448e96985e1b5b9fe023489c59ff746520a6ede: <one-sentence reason>

@bolichen97
bolichen97 force-pushed the fix/mcp-deferred-long-tools branch from ff5ae88 to 0408ccd Compare October 10, 2026 11:56
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 10, 2026
@bolichen97
bolichen97 force-pushed the fix/mcp-deferred-long-tools branch from 0408ccd to 57c05d2 Compare October 10, 2026 12:08
@github-actions

github-actions Bot commented Oct 10, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 9448e96985e1b5b9fe023489c59ff746520a6ede — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 9448e96

Verdict parsed from the review's SHA-scoped output markers for commit 9448e96985e1b5b9fe023489c59ff746520a6ede.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 9448e96985e1b5b9fe023489c59ff746520a6ede: <one-sentence reason>

@bolichen97
bolichen97 force-pushed the fix/mcp-deferred-long-tools branch 2 times, most recently from 2c3060b to 0b9011b Compare October 10, 2026 12:59
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 10, 2026
@bolichen97
bolichen97 force-pushed the fix/mcp-deferred-long-tools branch from 0b9011b to c9c9a2f Compare October 10, 2026 13:53
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 10, 2026
@bolichen97

Copy link
Copy Markdown
Contributor Author
  • The short ceiling is measured per call, but a call's real age includes its time in the queue span=861fe9d5644b

fixed in c9c9a2f. The in-flight report now also names every id still waiting in _pending_calls behind the worker (_queued_request_ids), and is sent whenever calls are parked or queued, so the gateway stamps t_progress_ms on a queued call for as long as it is queued. The call running on the worker is deliberately not named: its stamp freezes at about its start, so the 900s ceiling bounds one call's run, not the queue's length.
Tests: test_mcp_shared_deferred_tools.py::test_a_queued_call_is_reported_until_it_starts_running (queued call named, running call not, naming stops when it starts); test_mcp_gateway_inflight_report.py::test_calls_queued_behind_the_worker_do_not_age_the_backend_past_the_ceiling (five requests all past 900s of gateway age, four queued and reported: heartbeat says alive while the running one is at 170s, wedged once it runs past 900s).

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • Stamp t_progress_ms when the backend's worker actually starts a call span=c45f05f9b53c

fixed in c9c9a2f, by the complementary shape: the report names queued ids while they queue and stops naming a call when it starts running, so its last stamp IS its start. Naming the running id instead would refresh its stamp every 30s and a genuinely stuck worker would never cross the short ceiling; omitting it keeps the ceiling a bound on one call's run time, which is the property the suggestion asks for.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • Item 8 — zero consumers: is_tool_cancelled kept with no source caller span=476ca7308495

fixed in c9c9a2f. is_tool_cancelled, _thread_cancel_event, the worker's three set/clear sites and the stale comment are deleted; cancel_evt stays for the late-cancel suppression under the lock, and ToolCancelled stays because a step may raise it (its docstring no longer names the deleted function). test_mcp_gateway_wedge_ping_gate.py drops test_cancel_event_check and its suppression test no longer depends on the function. The PR description's compatibility section now states the removal instead of claiming the function stays.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 10, 2026
@bolichen97
bolichen97 force-pushed the fix/mcp-deferred-long-tools branch from c9c9a2f to abd2634 Compare October 10, 2026 14:53
@github-actions github-actions Bot removed the readiness: action required A blocking check or review needs attention label Oct 10, 2026
@bolichen97

Copy link
Copy Markdown
Contributor Author
  • Withdraw opened questions when the step pool cannot start span=82915c542db1

fixed in 316fcc0. The pool-start failure now refuses through _refuse_pool_start, which runs _cancel_quietly (the tool retracts what it opened: an _AskStep withdraws its card) before answering failed, the same shape as _refuse_table_full and _refuse_pruned; a call that acted still answers with its abandon text. test_a_pool_that_cannot_start_fails_only_that_call now asserts the refused call's cancel hook ran and the surviving call's did not.
self-added: yes

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • a cancel while the step is still queued lets the step run and POST /api/spawn after the client cancelled span=2ed88f864efd

fixed in 18452da. _cancel_deferred now tries entry.stepping.cancel() first: a step still queued behind busy pool threads is withdrawn and the call settles at once (entry removed, cancel hook run, cancelled audited, no response); only a step already running is marked so its text is dropped on return -- the shape the pruned-exit path already used. Test: test_cancel_while_a_step_is_queued_withdraws_it_before_it_runs (one worker held, second call's queued step never runs after its cancel).
self-added: yes

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • subagent.md says the refusal line is bounded to 240 while spawn.py sets 800 span=8f01e917c86d

fixed in 18452da. The spec now says 800, matching _SA_ERROR_MAX_CHARS.
self-added: yes

@bolichen97

Copy link
Copy Markdown
Contributor Author

Rebased onto origin/main 2a0400d (same-file overlap: .github/black-baseline.txt; one content conflict in src/kiro_crew/mcp_shared.py from #18979, resolved by porting its change -- any id-bearing request while the worker is busy is answered through _dispatch -- into this PR's busy branch; #18979's test passes). No other change since 18452da, on which GPT and Opus passed.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • The 15-minute recycle depends on an unenforced rule span=3eeac2d79e37

rebutted: the check it asks for already ships. test_mcp_gateway_inflight_report.py::test_ceiling_outlives_every_synchronous_tool_bound is an AST scan of every timeout= argument and *_TIMEOUT_SECS constant under mcp_tools/ and mcp_core.py (23 bounds today, max _POD_UP_TIMEOUT_S 210s) and fails when any synchronous kirocrew-core bound reaches PROGRESS_WEDGE_CEILING_SECS minus HEARTBEAT_TIMEOUT_SECS headroom; its docstring names the one gap (a tool that blocks with no named bound), which is a defect on main as well. Refreshing the running call's stamp is the alternative this PR rejects on purpose: it would let a hung worker age under the 3h ceiling again, the #17031 stall.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • The take-away check is capped by truncation span=7a54f230dadd

fixed (description only; no code change). The description was shortened by ~550 bytes in What changed, so ## Backwards compatibility now ends at byte ~7900 of the capture, inside the 8000-byte cut, with all four Reader lines, the SPAWN_BATCH_MEMBERS_MAX readers and the ASK_WAIT_SLICE_SECS readers intact. diff_signals --check-body and take_away_check.py pass on it (472 words).

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • Item 4: spawn_run deferral oversized span=57f42ff1d927

rebutted (recurring: 5th head). spawn_run submission is the same shared-worker hold the Goal names, in the handler module the Goal names: each member is one _post of up to 30s on the one worker, so a 10-task batch holds every other session's kirocrew-core call for up to 300s and a 128-task one crosses the 900s ceiling by itself -- the #17031 stall by another route, and a measurement any reader can make from _POST_TIMEOUT. The Goal is frozen and says one session's long call must not stall others. _SpawnRunStep is one class doing one POST per step; _close_unsubmitted is the only way to report members a cancel left unsubmitted; the cap bounds what the parked step retains, which the blocking rule a-bound-bounds-every-field-it-retains requires.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • Cancelled queued steps escape the retention bound span=82915c542db1

fixed in 9405fe5. A _StepPool queue item is now a small holder whose fn and args are released the moment its future is cancelled (Future.add_done_callback), so a cancelled step's call and payload leave the queue at cancel time and only an empty shell waits for the worker to drain it; the worker skips a shell. Test: TestStepPool::test_a_cancelled_queued_item_drops_its_payload_before_a_worker_reaches_it (weakref on the payload is dead after cancel, the step never runs).
self-added: yes

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • The short ceiling depends on a rule nothing checks span=4e51e0841b7e

rebutted: the test the clears-when asks for ships in this PR. test_mcp_gateway_inflight_report.py::test_ceiling_outlives_every_synchronous_tool_bound walks every module under mcp_tools/ plus mcp_core.py by AST, collects each timeout= argument and *_TIMEOUT_SECS/*_TIMEOUT_S constant (23 today, max _POD_UP_TIMEOUT_S = 210s) and fails when any reaches PROGRESS_WEDGE_CEILING_SECS - HEARTBEAT_TIMEOUT_SECS; a new synchronous handler with a long bound fails it before it is registered. It is named in the description's Tests section.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • Part of the compatibility list was cut off span=cce68b4ff044

fixed (description). The description was shortened by ~550 bytes before this head's review captured it; the capture for 9405fe5 ends ## Backwards compatibility at about byte 7900 with all four Reader lines inside the 8000-byte cut. The review re-runs on 9405fe5.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • Item 4: spawn_run deferral oversized span=43a1e6b1a442

rebutted (recurring, 6th head; see the disposition on 18452da for the measurement). spawn_run submission holds the one shared worker for one _post of up to 30s per member, so a batch holds every other session's kirocrew-core call for minutes and a 128-member batch alone crosses the 900s ceiling: the #17031 stall by another route, inside the module the frozen Goal names. The deferral is one class doing one POST per step; _close_unsubmitted is the only way to report members a cancel left unsubmitted; the cap bounds what the parked step retains, which the blocking rule requires.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • Item 4: spawn_run deferral oversized span=98a60763edf1

rebutted (recurring, 7th head; a read-only retrospective on this head judged it keep). spawn_run submission is one _post of up to 30s per member on the single shared worker: a 10-task batch holds every other session's kirocrew-core call for up to 300s, the #17031 stall by another route, in the module the frozen Goal names. _SpawnRunStep is one class doing one POST per step; _close_unsubmitted is the only way to report members a cancel left unsubmitted; the member cap bounds what the parked step retains, which the blocking rule a-bound-bounds-every-field-it-retains requires.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • Item 5: the 128-member cap rides along / is inherited retention span=075cbc508749

rebutted (recurring; a read-only retrospective on b5a1840 judged it keep). The retention is not the same as base's: base held ONE member list at a time, on the one blocking worker; a parked step holds its list while 63 other parked steps may hold theirs, so DEFERRED_PARKED_MAX bounds the count of lists and SPAWN_BATCH_MEMBERS_MAX bounds each one -- without the second bound the product is unbounded, which is what the AUTOSDE blocking rule a-bound-bounds-every-field-it-retains flagged when the cap was absent (GPT BLOCK-MERGE on an earlier head; the cap is what cleared it). 128 is twice the sub-agent ceiling, above any batch a caller can run at once, so no accepted call is refused in practice; the refusal is explicit, before any deferred state exists.

@bolichen97

Copy link
Copy Markdown
Contributor Author

Rebased onto origin/main 22a7358 (+70 commits). Two content conflicts resolved: website/scripts/check-bundle-size.mjs -- main landed the same App: 2200 * KB re-measure, so this PR's hunk is dropped and the file is no longer touched here; docs/system-specs/modules/subagent.md -- main's rewrite of the spawn_sub_agents bullets kept, with this PR's three wording changes (deadline starts after the last submission step; _ResumeHold.probe; the tool's own partial row beside the wrapper's completed row) re-applied. No code change since bd1e2dd; 1095 related tests pass locally; mypy count equals main's.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • Item 5: spawn_run deferral oversized span=ec687e7daccb

rebutted (recurring, 8th head; the round-24 read-only retrospective judged it keep). The harm is measurable from the code: each member of spawn_run is one _post of up to 30s on the ONE shared worker, so a 10-task batch holds every other session's kirocrew-core call for up to 300s -- the #17031 stall by another route, in the handler module the frozen Goal names. _SpawnRunStep is one class doing one POST per step; _close_unsubmitted is the only way to report members a cancel left unsubmitted; the bounded lost-report keeps the dispatch thread from blocking on a dead gateway.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • Item 6: the 128 cap on spawn_run.tasks/agents rides along span=d1e5fd0783dc

rebutted; falls with Item 5. A parked step holds its member list while up to 63 other parked steps hold theirs, so DEFERRED_PARKED_MAX bounds the number of lists and SPAWN_BATCH_MEMBERS_MAX bounds each; without the second the product is unbounded, which the blocking rule a-bound-bounds-every-field-it-retains flagged (GPT BLOCK-MERGE on an earlier head; this cap cleared it). 128 is twice the sub-agent ceiling, so no batch a caller can run at once is refused.

@buluoray buluoray left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes at eb2eab4 for one doc line only. Auto-merge is on, so this review is the last gate before main.

Blocking

  • src/kiro_crew/docs/agent-questions.md:52 (shipped user doc) still draws the wait loop as "20 s slices". This PR changes ASK_WAIT_SLICE_SECS from 20 to 5 and updates the system spec to match, so the user-facing doc now contradicts it. Please change it to 5 s in this PR.

While you are in there, follow-up 1 below is worth adding in the same push: it is the only guard on the cross-session path with no test.

Everything else holds up against the checks that matter for a shared backend:

  • Parked state is per caller. The parked table and step pool are locals of the dispatch loop; everything the PR adds at module level is a constant. Entries are keyed by the gateway's unique forward id, so a settled result goes back to the stub that made the call. Every step, cancel hook, abandon hook and settle commit runs with that call's own caller and tenant nonce, and clears them afterwards.
  • Every parked wait is bounded.
    • The parked table caps at DEFERRED_PARKED_MAX, with a retryable -32000 on overflow.
    • notifications/cancelled withdraws a queued step, or marks a running one and discards its result, then runs cancel() with no response.
    • A pruned-install exit lets an in-flight step finish within DEFERRED_STEP_STUCK_SECS before answering or refusing.
    • The loop's finally cancels queued futures and shuts the pool down.
    • On the gateway, a reporting backend recycles at PROGRESS_WEDGE_CEILING_SECS, and a cancelled entry is skipped only by the hard ceiling, so the two-condition ping rule still applies to it.
  • The wait early end is preserved. A keepalive goes out every WAIT_PING_SECS, a reply must name this wait_id to end the sleep, and the early end is a normal result rather than ToolCancelled.
  • Upgrade-safe. The report is advertised only by an opted-in server on POSIX (platform_compat.IS_POSIX), and only kirocrew-core opts in. The gateway lowers the ceiling only for a backend that advertised it. _call_tool and the Windows path still drive a deferred tool inline.

Guards checked by mutation: removing the cancel hook, the parked-table bound, the short ceiling, the cancelled-entry skip, the wait_id match or the mid-step cancel drop each turns a named test red.

Follow-ups (non-blocking except where noted above):

  1. No test pins that a deferred step runs under its own call's caller identity. Replacing set_current_caller(entry.caller_ctx) in _run_step with None leaves every touched suite green. A loop-level test should assert the identity a step sees, as test_spawn_collection_commit_installs_its_call_identity does for the settle commit. This is the guard that keeps a parked wait's keepalive on its own session in a pooled backend.
  2. Cancel hooks make bounded HTTP calls (5s each) on the dispatch thread. If a stub with many parked calls disconnects while the gateway is unresponsive, ping answers can be held for up to DEFERRED_PARKED_MAX × 5s. Consider moving the hooks onto the step pool, or capping the total hook time per tick.
  3. On EOF, parked calls' cancel() hooks are not run, so an open question card is left to the stale reaper. This matches the behaviour before this change, but it is worth a comment or a best-effort pass.

@bolichen97

Copy link
Copy Markdown
Contributor Author

Addressed at 4f3e422 (rebased onto main@5639ea7c, still one commit).

Blocking

  • src/kiro_crew/docs/agent-questions.md:52: (20 s slices) -> (5 s slices), matching ASK_WAIT_SLICE_SECS and the system spec.

Follow-up 1 (taken, same push)

  • New loop-level TestStepRunsUnderItsOwnIdentity in test/test_mcp_shared_deferred_tools.py: two sessions park a call each (dashboard:alpha / dashboard:beta, distinct tenant nonces); every step is asserted to see only its own call's caller and nonce, and a probe wrapped around the pool's submit asserts the thread holds (None, "") between steps. Mutation checked: replacing set_current_caller(entry.caller_ctx) in _run_step with None reds it.

Follow-ups 2 and 3: noted, not taken here. Both are pre-existing shapes (cancel hooks already ran on the dispatch thread for a worker-path cancel; EOF already left cards to the stale reaper) and sit outside the frozen goal. Happy to take them as a follow-up PR if you want them.

Also in this push: the ratchet (test_flake_pattern_ratchet.py, a local pre-push gate since CI checks out depth 1) flagged the 50 ms tick pace in test_mcp_gateway_wedge_ping_gate.py; it is a pace, not a bound (expiry only yields another tick), so it now carries # flake-ok with that reason.

The pooled kirocrew-core backend runs tool calls on one worker thread,
and wait, spawn_sub_agents and ask_question spent their whole duration
asleep on it. One session parked on a 1800s wait queued every other
session's call behind it, and because the loop still answered pings the
gateway's wedge detector held the backend healthy for the 3h ceiling.

A tool that only waits now returns a DeferredTool: setup runs on the
worker, the waiting is short steps the dispatch loop times from its own
tick and runs on a small pool, so the worker is free the moment the
handler returns and a thousand parked sessions hold no thread. The loop
reports its parked ids to a gateway that asked for them in initialize;
the gateway ages those requests from the last report and holds a
reporting backend to a 900s ceiling instead of 3h, so a genuinely hung
call recycles in minutes.

Closes #17031
@bolichen97

Copy link
Copy Markdown
Contributor Author
  • The short ceiling rests on an unwritten rule span=20bb197c0a3e

rebutted (recurring, 4th head on this rule; covers the Cause/Mechanism/Consequence/Clears-when spans 1d417af08bc9, 4583a6ded186, d9c79108e5b3, cb3f2d1f093b, d14298fd60c2 of the same item). The check the clears-when asks for ships in this PR and is named in the description's Tests section: test_mcp_gateway_inflight_report.py::test_ceiling_outlives_every_synchronous_tool_bound walks every module under mcp_tools/ plus mcp_core.py by AST, collects each timeout= argument and *_TIMEOUT_SECS/*_TIMEOUT_S constant (23 today, max _POD_UP_TIMEOUT_S = 210s) and fails when any reaches PROGRESS_WEDGE_CEILING_SECS - HEARTBEAT_TIMEOUT_SECS, so a future synchronous tool with a longer bound fails CI before it ships rather than recycling production backends. Refreshing the running call's stamp is rejected on purpose: it would age a hung worker under the 3h ceiling again, which is the #17031 stall.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • Suggestion: a SYNC_TOOL_MAX_SECS constant the ceiling derives from span=55eb8e63062c

rebutted (not taken). A hand-maintained constant is a second copy of a fact the AST scan already measures from the tools themselves; the scan fails on the real bound a tool declares, a constant fails only when someone remembers to raise it. Per-tool asserts against the constant would add one line per long-bounded tool with no new guarantee beyond the existing test. Not a mechanism the frozen goal needs.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • Item 4: spawn_run deferral oversized span=38f6ae95cc73

rebutted (recurring, 9th head; the round-24 read-only retrospective judged it keep). The Goal's sentence is about one session's long kirocrew-core call stalling other sessions' calls; spawn_run submission is exactly that hold: each member is one _post of up to 30s on the ONE shared worker, so a 10-task batch holds every other session's call for up to 300s and a 128-task one crosses the 900s ceiling by itself -- the #17031 stall by another route, in the handler module the Goal names. _SpawnRunStep is one class doing one POST per step; _close_unsubmitted is the only way to report members a cancel left unsubmitted; the bounded lost-report keeps the dispatch thread off a dead gateway.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • Item 8: the must be a string refusal rides along span=57006b665e05

rebutted. On base a non-string prompt/agent_or_mode failed inside the blocking handler's own len()/slice on the worker and was gone with the call. In this PR the member list is RETAINED by the parked _SubAgentsStep until its submit step posts it, and the length clamp that bounds a string does not bound a dict or list, so a non-string member would sit unbounded in the parked table -- the retention this PR introduces, flagged by a GPT BLOCK on 6551be9. The refusal is the smallest guard for that retention (two isinstance lines, no new mechanism); test test_mcp_core_spawn_sub_agents.py pins it. It is a defect in code this PR adds, which the frozen-goal rule keeps in scope.

@buluoray buluoray left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve at 9448e96985. This supersedes my earlier request changes at eb2eab454b.

What this brings users: one session's wait or ask_question no longer stalls other sessions' tools on the shared MCP worker, and a stuck backend is recycled in about 15 minutes instead of 3 hours.

Both asks from my last review are done

  • Doc: src/kiro_crew/docs/agent-questions.md:52 now says 5 s slices, matching the new slice.
  • Identity test: TestStepRunsUnderItsOwnIdentity parks two calls from two sessions and checks every step sees its own caller and tenant nonce, and that the pool thread is cleared between steps.
    • Mutation: making _run_step (src/kiro_crew/mcp_shared.py:2725-2738) install no caller turns test_each_step_sees_its_call_identity_and_the_pool_thread_is_cleared red. Restored, the file passes (49 tests).

Range-diff

Besides the two items above, the only change is in test_mcp_gateway_wedge_ping_gate.py: the 0.05 s wait is now documented as a tick pace with the tick count as the real bound. No product code changed.

CI on this head: 89 success, 9 skipped, 0 failure; legacy statuses (Linux, Windows, PR Readiness) all success.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

3 participants