Repository navigation
fix(mcp): stop conductor patrol reads timing out on a busy gateway - #21423
KiroCrewCaptain merged 1 commit into
Conversation
5bfa514 to
4d1e15d
Compare
4d1e15d to
b64b5ca
Compare
|
Rebased onto main to clear a conflict: head is now b64b5ca. Behaviour is unchanged. #21524 split
Retested on the new head: the 78 test files that exercise the changed modules pass (4,049 passed, 1 skipped), including the facade and mirror guards. All 12 mutations listed in the description still fail at least one test. black, isort, flake8, mypy, the comment-history and brand checks, file line cap, docs-lint and the flake ratchet pass. The description's "What changed" and "Manual verification" sections now name the new locations. |
UX Review (Opus 5.5, fork) — ⏭️ skippedRevision |
Design Review (Opus 5.5, fork) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing blocks. The change finds the measured cause (queue wait paid once per hop), cuts the field list down to one hop, and gives both reads the same 30 s budget their sibling writes already get. [DESIGN-REVIEWED] ece0577 Description read[DESCRIPTION-READ] bf699db471c28817876cb57f10dbc1b4ba7f63c4381df91c97e44aa495f727dc That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5, fork) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS The 30 s budgets fix the symptom, not the cause, which is executor queue wait. The PR's Notes
What this change shipsInventory (4 items) — 3 justifiedIntent: a conductor's patrol reads (session roster, field list) answer on a busy gateway instead of timing out at 10 s. This is a FIX.
[FIRST-PRINCIPLES-REVIEWED] ece0577 Description read[DESCRIPTION-READ] f52024388414dd05ef522614f07253f4e4b784194ca43b9be1a55e6dba00d22f That sha256 names the PR description this verdict read. Recompute it with |
GPT 6.1 Review (fork) — ✅ no blocking findingsReviewed Review detailsNo findings. |
Opus 5.5 Review (fork) — ✅ no blocking findingsReviewed |
session_status and dashboard_fields failed at mcp_core._get's 10 s telemetry default for a busy conductor while its session_send calls, on _post's 30 s, kept landing. Both GETs do real work on the gateway's shared default executor, where every worker-thread hop waits for the queue before it starts. Give both tools the write path's 30 s budget, the clear path's type lookup included, and run the field list's five reads in one worker job so a request pays one queue wait instead of five.
b64b5ca to
ece0577
Compare
|
Pushed
|
|
First Principles on |
KiroCrewCaptain
left a comment
There was a problem hiding this comment.
Tier 1 auto-approve: fix (5 files). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean, security checklist all-NO, no reviewer-lane findings, readiness signal fresh for this head SHA, declared category matches the diff. Category: raises the session_status and dashboard-fields read timeouts to the 30 s write-path value and folds the dashboard-fields reads into one worker hop, so conductor patrol reads stop timing out on a busy gateway; tests added.
Problem / Motivation
Goal: A busy conductor's
session_statusanddashboard_fieldscalls answer instead of failing at the MCP client's 10 s GET bound.On one host, a crewmate conductor's two patrol reads returned
execution_failedat about 10.1 s while it was dispatching workers:@kirocrew-dashboard/session_statusfailed once at 10.08 s (and completed once at 9.45 s), and@kirocrew-panel/dashboard_fieldsfailed three times at 10.11 to 10.14 s, all inside seven minutes. The gateway logged no handler error for any of them. Eachdashboard_fieldscall wrote itsapi_access dashboard_fields allowedaudit about 0.17 s after the invoke, so the time went into the route after its gate. The same conductor'ssession_sendandsession_read_messageanswered in 0.1 to 1.3 s throughout, so the event loop was responsive, and in the same minutes a 3 s thread-hop read elsewhere in the gateway timed out (Process tree refresh ... timed out after 3.0s).Both tools call
mcp_core._get, whose 10 s default is sized for a telemetry read. Both routes do real work on the gateway's shared default executor, where everyasyncio.to_threadhop waits its turn in the queue before it starts.Why it matters
These are the reads a conductor's patrol makes every cycle. The roster says which workers are working, idle, closed or lost, and the field list is the read every dashboard write starts from. They fail when the conductor is busiest, which is when it has the most workers to account for.
Error: timed outnames no budget, so the failure reads as a gateway fault rather than a client bound that is too small, while the conductor's writes keep landing on their 30 s budget.Not a goal
executors.pyrecords, and is not this change.api_accessaudits are unchanged.read_slot_projection(slot, "status")lists the store's units again; batching that is a crew-log change._current_values), which is most of that route's own work.What changed (motivation → approach → change)
Measured first. A local load test drove the real route handlers on an instrumented default executor of the gateway's size (32 workers), recording each
asyncio.to_threadhop as queue wait (submit to start) and work (start to end). Fixture: a conductor with 250 ended workers, each with a 20-turn crew log, 1,700 transcripts in the catalog, and a crewmate DM slot with 40 logs. "Saturated" keeps a backlog of blocking 0.25 s jobs queued ahead of every hop: depth 64 gives about 0.5 s of queue wait per hop, depth 160 about 1.25 s.The extra hop in some rows is the SEL audit write, offloaded when its writer is not warm.
What the numbers say:
_gone_slot_lifecyclesfolds one crew log per ended child (about 3.6 ms each warm and 15 ms cold at this log size), and only for children inside the roster's 256-row cap (MAX_SESSION_STATUS_ROWS), so 250 ended children cost 0.90 s warm and 3.80 s cold. Its hops stay: two are caller gates, and the history scan and the ended-row fold are separated by caller re-checks and a live read that must run on the loop. This PR leaves the route as it is on main and changes only the client's budget for it.Changes:
session_ledger_readprecedent did it:session_statusanddashboard_fieldspass 30 s tomcp_core._get, the budget their sibling writes (session_send,dashboard_write) already get from_post. Theclear: truetype lookupdashboard_writemakes against the fields route uses the same constant. Each constant lives beside its one consumer, the waysession_ledger_read's_LEDGER_READ_TIMEOUT_Sdoes:SESSION_STATUS_READ_TIMEOUT_SECSbeside_run_session_statusinmcp_dashboard_session_runs.py, bound on themcp_dashboardfacade like every name that module defines, andDASHBOARD_FIELDS_READ_TIMEOUT_SECSinmcp_panel.py.api_dashboard_fields(dashboard/handlers/agent_panel_dashboard.py) runsread_instance,_mistake_book,_instance_history,_current_valuesandfields_for_agentin one worker job. They were already sequential, each later read needing the instance the first returns, and the payload is unchanged. At about 1.25 s per hop the route goes from 9 hops and 10.79 s to 5 hops and 5.94 s.At this fixture's work, the 30 s budgets let the field list absorb about 5.8 s of queue wait per hop and the roster about 7.3 s warm or 6.5 s with cold folds. With main's 10 s budget the field list broke at about 1.0 s per hop, and the roster at about 2.3 s warm or 1.5 s with cold folds. A crewmate caller pays one more gate hop on the roster than the fixture's plain chat caller.
Backwards compatibility
Compatible: both client budgets only widen, the roster route is unchanged, and the field list returns the same payload from the same reads in the same order.
Removes nothing: the diff widens two client timeouts and moves five sequential reads into one worker job; no row, field, key, function or accepted input goes away.
Tests
test/test_conductor_read_budgets.py(new):test_session_status_reads_a_roster_slower_than_the_telemetry_budgetandtest_dashboard_fields_reads_a_field_list_slower_than_the_telemetry_budget: the real MCP tool andmcp_coretransport, with only the socket faked as a 12 s route, answer instead of timing out.test_a_clear_reads_the_field_types_on_the_same_budget:clear: trueon an object field still resolves{}against that slow route instead of falling back to[], which the write route would refuse aswrong_typeand book as the agent's mistake.test_both_read_budgets_match_the_write_path: both budgets are at least_send's default and above_get's, read off the transport.test_the_field_list_pays_one_queue_wait_for_all_its_reads: on a default executor that counts what is submitted to it, the route makes ONE submission for its five reads, all of them on one worker thread and none on the loop.test/test_session_status.pyis unchanged, because the roster route is main's, and it passes on this head, including its 12 tests of closed and lost rows: a worker that closed its tab isclosedwith its stamp; one gone with a live log, reset, destroyed, or closed mid-turn islost; a read fault on a gone slot defaults tolost. The guards of the two split module families,test/test_mcp_dashboard_facade.py(every name an owner defines is bound on the facade) andtest/test_agent_panel_facade.py, pass with the new constant bound.Mutation check: each of 6 source mutations turns at least one of these tests red: dropping each of the three client timeouts (roster, field list, the
clearlookup), setting the roster's budget to 11 s, splitting one field-list read into its own hop, and running the field-list reads on the loop.Manual verification
The load test above, run against
origin/mainat this PR's original basec1302956a1and against this PR's first head5bfa514b86, with the same fixture and load levels. The field-list change it measured is the one this head ships: main has since movedapi_dashboard_fieldsfromagent_panel.pyintoagent_panel_dashboard.py(#22026), and the rebase carried the change over unaltered. This head's roster route is main's, so the status rows were measured on main. Not reproduced on a live gateway under the original load.Local gates on the current head, rebased onto main
3d6cda4f3a: black, isort, flake8, mypy, subprocess encoding, comment-history, brand, harness parity, memory-store seam, feature map, file line cap, sync-io-in-async, loop-bound locks, docs-lint andtest/test_flake_pattern_ratchet.pypass, the 98 test files that exercise the changed modules and their facade guards pass (4,755 passed, 46 skipped), and each of the 6 mutations above is caught. The related-set backend test gate selects over 600 files; the full suite is left to CI. The frontend gates were not run: the diff touches nothing underwebsite/.Related Issues
Precedent: #16663 / #16665 raised
session_ledger_read's client budget for the same reason. No open issue or PR covers these two tools.Pattern harvest
Rule candidate: review-prompt
Pattern: a route handler that awaits several sequential
asyncio.to_threadhops for one reply, each paying the shared default executor's queue wait, or an MCP tool whose GET does disk or fold work and keepsmcp_core._get's 10 s telemetry default.Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)Contribution License Agreement
Contributions are accepted under the project's Apache-2.0 license; see the Licensing section of CONTRIBUTING.md.