Skip to content

fix(mcp): stop conductor patrol reads timing out on a busy gateway - #21423

Merged
KiroCrewCaptain merged 1 commit into
kirodotdev:mainfrom
rubencu:fix/conductor-read-tools-timeout-bound
Oct 22, 2026
Merged

KiroCrewCaptain merged 1 commit into
kirodotdev:mainfrom
rubencu:fix/conductor-read-tools-timeout-bound

Conversation

@rubencu

@rubencu rubencu commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A busy conductor's session_status and dashboard_fields calls answer instead of failing at the MCP client's 10 s GET bound.

On one host, a crewmate conductor's two patrol reads returned execution_failed at about 10.1 s while it was dispatching workers: @kirocrew-dashboard/session_status failed once at 10.08 s (and completed once at 9.45 s), and @kirocrew-panel/dashboard_fields failed three times at 10.11 to 10.14 s, all inside seven minutes. The gateway logged no handler error for any of them. Each dashboard_fields call wrote its api_access dashboard_fields allowed audit about 0.17 s after the invoke, so the time went into the route after its gate. The same conductor's session_send and session_read_message answered in 0.1 to 1.3 s throughout, so the event loop was responsive, and in the same minutes a 3 s thread-hop read elsewhere in the gateway timed out (Process tree refresh ... timed out after 3.0s).

Both tools call mcp_core._get, whose 10 s default is sized for a telemetry read. Both routes do real work on the gateway's shared default executor, where every asyncio.to_thread hop waits its turn in the queue before it starts.

Why it matters

These are the reads a conductor's patrol makes every cycle. The roster says which workers are working, idle, closed or lost, and the field list is the read every dashboard write starts from. They fail when the conductor is busiest, which is when it has the most workers to account for. Error: timed out names no budget, so the failure reads as a gateway fault rather than a client bound that is too small, while the conductor's writes keep landing on their 30 s budget.

Not a goal

  • Making the shared default executor less contended. Over 1,700 call sites route onto it; moving work off it is the direction executors.py records, and is not this change.
  • Removing the caller gates' own hops. The fields route's gate keeps its three hops and the roster keeps its scope and prewarm hops. Auth gates and api_access audits are unchanged.
  • Making one status fold cheaper. Each read_slot_projection(slot, "status") lists the store's units again; batching that is a crew-log change.
  • Speeding up the field list's values read (_current_values), which is most of that route's own work.

What changed (motivation → approach → change)

Measured first. A local load test drove the real route handlers on an instrumented default executor of the gateway's size (32 workers), recording each asyncio.to_thread hop as queue wait (submit to start) and work (start to end). Fixture: a conductor with 250 ended workers, each with a 20-turn crew log, 1,700 transcripts in the catalog, and a crewmate DM slot with 40 logs. "Saturated" keeps a backlog of blocking 0.25 s jobs queued ahead of every hop: depth 64 gives about 0.5 s of queue wait per hop, depth 160 about 1.25 s.

route load hops queue work total, median of 5
fields, main idle, warm 8 0.00 s 0.94 s 0.94 s
fields, main ~0.5 s per hop 9 3.92 s 0.95 s 4.82 s
fields, main ~1.25 s per hop 9 9.96 s 0.96 s 10.79 s (max 11.00 s)
fields, this PR ~0.5 s per hop 5 1.85 s 0.95 s 2.80 s
fields, this PR ~1.25 s per hop 5 4.98 s 0.95 s 5.94 s
status, main and this PR idle, cold folds 4 0.00 s 4.12 s 4.12 s
status, main and this PR idle, warm 5 0.00 s 0.97 s 0.97 s
status, main and this PR ~1.25 s per hop 4 6.01 s 0.99 s 5.75 s

The extra hop in some rows is the SEL audit write, offloaded when its writer is not warm.

What the numbers say:

  • fields reproduces the failure: at about 1.25 s per hop the route on main crosses 10 s, and 92% of it is queue wait. Its five reads after the gate are five sequential hops, so it pays the queue five times for about 1 s of work.
  • status is queue-dominated too, and its work is already bounded on main: _gone_slot_lifecycles folds one crew log per ended child (about 3.6 ms each warm and 15 ms cold at this log size), and only for children inside the roster's 256-row cap (MAX_SESSION_STATUS_ROWS), so 250 ended children cost 0.90 s warm and 3.80 s cold. Its hops stay: two are caller gates, and the history scan and the ended-row fold are separated by caller re-checks and a live read that must run on the loop. This PR leaves the route as it is on main and changes only the client's budget for it.

Changes:

  1. Client budgets, the way the session_ledger_read precedent did it: session_status and dashboard_fields pass 30 s to mcp_core._get, the budget their sibling writes (session_send, dashboard_write) already get from _post. The clear: true type lookup dashboard_write makes against the fields route uses the same constant. Each constant lives beside its one consumer, the way session_ledger_read's _LEDGER_READ_TIMEOUT_S does: SESSION_STATUS_READ_TIMEOUT_SECS beside _run_session_status in mcp_dashboard_session_runs.py, bound on the mcp_dashboard facade like every name that module defines, and DASHBOARD_FIELDS_READ_TIMEOUT_SECS in mcp_panel.py.
  2. One hop for the field list. api_dashboard_fields (dashboard/handlers/agent_panel_dashboard.py) runs read_instance, _mistake_book, _instance_history, _current_values and fields_for_agent in one worker job. They were already sequential, each later read needing the instance the first returns, and the payload is unchanged. At about 1.25 s per hop the route goes from 9 hops and 10.79 s to 5 hops and 5.94 s.

At this fixture's work, the 30 s budgets let the field list absorb about 5.8 s of queue wait per hop and the roster about 7.3 s warm or 6.5 s with cold folds. With main's 10 s budget the field list broke at about 1.0 s per hop, and the roster at about 2.3 s warm or 1.5 s with cold folds. A crewmate caller pays one more gate hop on the roster than the fixture's plain chat caller.

Backwards compatibility

Compatible: both client budgets only widen, the roster route is unchanged, and the field list returns the same payload from the same reads in the same order.

Removes nothing: the diff widens two client timeouts and moves five sequential reads into one worker job; no row, field, key, function or accepted input goes away.

Tests

test/test_conductor_read_budgets.py (new):

  • test_session_status_reads_a_roster_slower_than_the_telemetry_budget and test_dashboard_fields_reads_a_field_list_slower_than_the_telemetry_budget: the real MCP tool and mcp_core transport, with only the socket faked as a 12 s route, answer instead of timing out.
  • test_a_clear_reads_the_field_types_on_the_same_budget: clear: true on an object field still resolves {} against that slow route instead of falling back to [], which the write route would refuse as wrong_type and book as the agent's mistake.
  • test_both_read_budgets_match_the_write_path: both budgets are at least _send's default and above _get's, read off the transport.
  • test_the_field_list_pays_one_queue_wait_for_all_its_reads: on a default executor that counts what is submitted to it, the route makes ONE submission for its five reads, all of them on one worker thread and none on the loop.

test/test_session_status.py is unchanged, because the roster route is main's, and it passes on this head, including its 12 tests of closed and lost rows: a worker that closed its tab is closed with its stamp; one gone with a live log, reset, destroyed, or closed mid-turn is lost; a read fault on a gone slot defaults to lost. The guards of the two split module families, test/test_mcp_dashboard_facade.py (every name an owner defines is bound on the facade) and test/test_agent_panel_facade.py, pass with the new constant bound.

Mutation check: each of 6 source mutations turns at least one of these tests red: dropping each of the three client timeouts (roster, field list, the clear lookup), setting the roster's budget to 11 s, splitting one field-list read into its own hop, and running the field-list reads on the loop.

Manual verification

The load test above, run against origin/main at this PR's original base c1302956a1 and against this PR's first head 5bfa514b86, with the same fixture and load levels. The field-list change it measured is the one this head ships: main has since moved api_dashboard_fields from agent_panel.py into agent_panel_dashboard.py (#22026), and the rebase carried the change over unaltered. This head's roster route is main's, so the status rows were measured on main. Not reproduced on a live gateway under the original load.

Local gates on the current head, rebased onto main 3d6cda4f3a: black, isort, flake8, mypy, subprocess encoding, comment-history, brand, harness parity, memory-store seam, feature map, file line cap, sync-io-in-async, loop-bound locks, docs-lint and test/test_flake_pattern_ratchet.py pass, the 98 test files that exercise the changed modules and their facade guards pass (4,755 passed, 46 skipped), and each of the 6 mutations above is caught. The related-set backend test gate selects over 600 files; the full suite is left to CI. The frontend gates were not run: the diff touches nothing under website/.

Related Issues

Precedent: #16663 / #16665 raised session_ledger_read's client budget for the same reason. No open issue or PR covers these two tools.

Pattern harvest

Rule candidate: review-prompt
Pattern: a route handler that awaits several sequential asyncio.to_thread hops for one reply, each paying the shared default executor's queue wait, or an MCP tool whose GET does disk or fold work and keeps mcp_core._get's 10 s telemetry default.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributions are accepted under the project's Apache-2.0 license; see the Licensing section of CONTRIBUTING.md.

@rubencu
rubencu requested review from a team and KiroCrewCaptain as code owners October 16, 2026 23:15
@rubencu
rubencu requested a review from chenmingwei23 October 16, 2026 23:15
@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: action required A blocking check or review needs attention merge conflict Branch has merge conflicts with its base — author must resolve before merge labels Oct 16, 2026
@rubencu
rubencu force-pushed the fix/conductor-read-tools-timeout-bound branch from 5bfa514 to 4d1e15d Compare October 17, 2026 05:53
@github-actions github-actions Bot added merge conflict Branch has merge conflicts with its base — author must resolve before merge and removed merge conflict Branch has merge conflicts with its base — author must resolve before merge labels Oct 17, 2026
@rubencu
rubencu force-pushed the fix/conductor-read-tools-timeout-bound branch from 4d1e15d to b64b5ca Compare October 17, 2026 17:51
@rubencu

rubencu commented Oct 17, 2026

Copy link
Copy Markdown
Contributor Author

Rebased onto main to clear a conflict: head is now b64b5ca. Behaviour is unchanged.

#21524 split validation.py into validation_parts owner modules, and test_validation_parts_facade.py requires those modules to define exactly the names frozen when it was split. So the two constants this PR added to validation.py now sit beside their single consumer instead of in a validation_parts module:

  • SESSION_STATUS_READ_TIMEOUT_SECS is in mcp_dashboard.py, next to _run_session_status. This matches DASHBOARD_FIELDS_READ_TIMEOUT_SECS in mcp_panel.py and _LEDGER_READ_TIMEOUT_S in mcp_tools/ledger.py.
  • MAX_SESSION_STATUS_GONE_ROWS is in session_control_parts/session_status.py, beside that module's other read bounds, and the session_control facade serves it.

validation.py and validation_parts/input_bounds.py are byte-identical to main.

Retested on the new head: the 78 test files that exercise the changed modules pass (4,049 passed, 1 skipped), including the facade and mirror guards. All 12 mutations listed in the description still fail at least one test. black, isort, flake8, mypy, the comment-history and brand checks, file line cap, docs-lint and the flake ratchet pass. The description's "What changed" and "Manual verification" sections now name the new locations.

@github-actions github-actions Bot removed the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Oct 17, 2026
@github-actions

github-actions Bot commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5, fork) — ⏭️ skipped

Revision ece057746708b81c11899c77e1021d59a4c2f61f touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 17, 2026
@github-actions

github-actions Bot commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5, fork) — ✅ PASS

Design-level review of ece057746708b81c11899c77e1021d59a4c2f61f via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks. The change finds the measured cause (queue wait paid once per hop), cuts the field list down to one hop, and gives both reads the same 30 s budget their sibling writes already get.

[DESIGN-REVIEWED] ece0577

Description read

[DESCRIPTION-READ] bf699db471c28817876cb57f10dbc1b4ba7f63c4381df91c97e44aa495f727dc

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5, fork) — ✅ PASS

Premise-level review of ece057746708b81c11899c77e1021d59a4c2f61f via the fork AI-review pipeline — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

The 30 s budgets fix the symptom, not the cause, which is executor queue wait. The PR's ## Not a goal names that cause and puts it out of scope, so this is a note, not a block.

Notes

  • Item 1 — symptom-level: "Making the shared default executor less contended … is not this change". The budget hides the queue wait on the roster route and does not remove it. The field list's single hop (item 2) is the mechanism-level fix, and the roster route has no equivalent.

What this change ships

Inventory (4 items) — 3 justified

Intent: a conductor's patrol reads (session roster, field list) answer on a busy gateway instead of timing out at 10 s. This is a FIX.

  1. session_status and dashboard_fields wait 30 s instead of 10 s before reporting a timeout — symptom-level: widens the bound, and the queue cause is declared out of scope
  2. The field list's five reads after the gate run in one worker hop instead of five, so a busy gateway charges the queue wait once — justified
  3. clear: true type lookup gets the same 30 s budget, so an object field no longer falls back to [] — justified
  4. SESSION_STATUS_READ_TIMEOUT_SECS bound on the mcp_dashboard facade — justified

[FIRST-PRINCIPLES-REVIEWED] ece0577

Description read

[DESCRIPTION-READ] f52024388414dd05ef522614f07253f4e4b784194ca43b9be1a55e6dba00d22f

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review (fork) — ✅ no blocking findings

Reviewed ece057746708b81c11899c77e1021d59a4c2f61f via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] ece0577

@github-actions

github-actions Bot commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review (fork) — ✅ no blocking findings

Reviewed ece057746708b81c11899c77e1021d59a4c2f61f via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] ece0577

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention merge conflict Branch has merge conflicts with its base — author must resolve before merge and removed readiness: checking Automated validation is still running labels Oct 17, 2026
session_status and dashboard_fields failed at mcp_core._get's 10 s
telemetry default for a busy conductor while its session_send calls,
on _post's 30 s, kept landing. Both GETs do real work on the gateway's
shared default executor, where every worker-thread hop waits for the
queue before it starts.

Give both tools the write path's 30 s budget, the clear path's type
lookup included, and run the field list's five reads in one worker job
so a request pays one queue wait instead of five.
@rubencu
rubencu force-pushed the fix/conductor-read-tools-timeout-bound branch from b64b5ca to ece0577 Compare October 22, 2026 02:52
@rubencu

rubencu commented Oct 22, 2026

Copy link
Copy Markdown
Contributor Author

Pushed ece0577467, this round's one push. It removes the ended-row bound, and it is rebased onto main 3d6cda4f3a to clear the conflict: #21510 and #22026 split mcp_dashboard.py and agent_panel.py into owner modules. The roster budget now sits beside _run_session_status in mcp_dashboard_session_runs.py, bound on the mcp_dashboard facade, and the one-hop field read is in agent_panel_dashboard.py. The move changes no behaviour. One test change came with the rebase: since #21191 a crewmate that adopted nothing is shown a page only when a shipped one is named for it, so the one-hop test reads kirocrew-conductor's board instead of an unnamed crewmate's default page.

  • First Principles, items 4 and 6 (the ended-row bound is oversized; the _reported_cut rider is undeclared): addressed by removing both. MAX_SESSION_STATUS_GONE_ROWS, the older-ended cut, gone_omitted (route, audit, tool caveat, docs, tests), _reported_cut and the newest-first tree sort are gone. The roster's route, its docs and test/test_session_status.py are main's, and the session_status tool differs from main only by its 30 s budget. The bound's premise was wrong: main already folds only the ended children inside the roster's 256-row cap (MAX_SESSION_STATUS_ROWS), so that work is bounded on main. The description no longer claims unbounded growth, and its load-test table shows only the shipped change.
  • Opus, older_ended drawn from the 256-capped tree_children, so older children could come back as unknown and gone_omitted undercounts: moot, that code is removed.
  • Design, the check that a conductor with more than 64 ended workers can still find an old lost one: moot, no ended-row bound remains. Ended rows are listed exactly as on main, inside its 256-row roster cap, and the closed and lost tests in test/test_session_status.py pass unchanged.
  • The red on b64b5cae90, Backend Tests (3.12, 7), was test/test_session_allocation_bindings.py::test_every_name_patched_through_the_facade_reaches_every_owner_reading_it hitting the 120 s per-test timeout: test_session_allocation_bindings facade-patch tree scan takes ~60s and times out at 120s on CI #21441, fixed on main by test: read only the facade's own files in the session allocation patch scan #21640, which this rebase brings in. Coverage Gate and PR Readiness failed only because that shard did.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 22, 2026
@rubencu

rubencu commented Oct 22, 2026

Copy link
Copy Markdown
Contributor Author

First Principles on ece0577467, the note on item 1 (the 30 s budgets are symptom-level): deferred, as the description's Not a goal says. Making the shared default executor less contended is the cause-level change, and it is out of scope here. The roster gets no one-hop change like the field list's because its remaining hops are not one sequential read: two are caller gates, and the history scan and the ended-row fold are separated by caller re-checks and a live read that must run on the loop. What the budget buys is under What changed: at the load test's work the roster absorbs about 7.3 s of queue wait per hop warm, against about 2.3 s on main's 10 s, and its read gets the 30 s its sibling session_send already has.

@github-actions github-actions Bot removed merge conflict Branch has merge conflicts with its base — author must resolve before merge readiness: checking Automated validation is still running labels Oct 22, 2026
@github-actions github-actions Bot added the readiness: passed Eligible automated validation passed for the current revision label Oct 22, 2026
@KiroCrewCaptain
KiroCrewCaptain enabled auto-merge (squash) October 22, 2026 06:03

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tier 1 auto-approve: fix (5 files). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean, security checklist all-NO, no reviewer-lane findings, readiness signal fresh for this head SHA, declared category matches the diff. Category: raises the session_status and dashboard-fields read timeouts to the 30 s write-path value and folds the dashboard-fields reads into one worker hop, so conductor patrol reads stop timing out on a busy gateway; tests added.

@KiroCrewCaptain
KiroCrewCaptain merged commit 30b2797 into kirodotdev:main Oct 22, 2026
102 checks passed
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants