Repository navigation
fix: move the spawn gate on its own evidence, not gateway loop lag - #17231
Conversation
First Principles Review (Opus 5.5) — 🟡 CONCERNSPremise-level review of First-Principles-Verdict: CONCERNS Two riders add new ways to cut the gate, but the incident had zero failures and no slow inits; ignoring lag plus idle recovery already fixes #17206. Not justified as shipped
What this change shipsInventory (6 items) — 4 justifiedIntent: let new chats on an idle host get their MCP tools instead of timing out. This is a FIX.
[FIRST-PRINCIPLES-REVIEWED] d4519cf Description read[DESCRIPTION-READ] bae931ef4e9d86b75c13b216c6b6174f4ab558cc874d08e4e6dec3acc68bd796 That sha256 names the PR description this verdict read. Recompute it with |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Check that The timer starts at [DESIGN-REVIEWED] d4519cf Description read[DESCRIPTION-READ] 3515b0c86114d8548009beeb93675d321af3ba329bd5b6931c7a4118a28fe232 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
Pull request was converted to draft
|
Holding this as a draft. #17017 rewrites The two parts that do not touch the policy (free a slot after 2 s with no |
The spawn gate admits forks inside the MCP daemon, but one loop-lag sample from the dashboard gateway was enough to cut it, and severe lag paused it. On an idle host with zero spawn failures it sat at 1 all day, backends started one at a time and new chats missed the MCP readiness deadline. The gate track now ignores loop lag. It cuts on critical memory, on gate init failures, or on two slow inits (each at 80% of its own initialize deadline, within 60 s), each sufficient alone, and pauses only on critical memory. A cut gate climbs back one step per clean window to its fresh-start value without needing queued demand. The exec track is unchanged. Closes #17206
d0b346f to
d4519cf
Compare
|
Rebased onto main now that #17017 has landed, and narrowed to the gate track only: the gate stops acting on the gateway loop lag and moves on its own evidence (critical memory, gate init failures, slow inits relative to each init's own deadline), with idle recovery. The exec track #17017 shaped is untouched. The teardown fix is in #17251; the 2 s early slot release is dropped. |
|
|
Problem / Motivation
Goal: New chats get their MCP tools on an idle host instead of failing the 90 s readiness timeout.
On one host (2026-10-05) gatewayd's spawn gate sat at 1 of 8 with 0 spawn failures out of 621, load 0.09 per core and 40 GB free. It was cut to 1 49 times that day, each time on
corroborated pressure: loop_lagfrom the dashboard gateway process. With the gate at 1, backends start one at a time and each holds the slot for up to 10 s (95 of 621 permits held the wholeinitialize_timeout_secswith noinitialize). kirocrew-core / kirocrew-work queue behind external MCPs, the client detaches its stub after ~30 s, gatewayd kills the still-starting backend (recycled (killed): last stub detached with in-flight work) and the retry goes to the back of the queue, until the chat times out. See #17206.Why it matters
Every new chat on an affected host can come up with no MCP tools, on a machine that is doing nothing. One dashboard event-loop hiccup is enough to start it, and the gate does not recover on its own.
Not a goal
initialize_timeout_secs.What changed (motivation → approach → change)
The spawn gate stopped reading the dashboard gateway's loop lag and now moves on its own evidence; a cut gate climbs back by itself.
loop_lagalone cut the gate; lag over 2 s paused itloop_lagnever moves the gate (GATE_IGNORED_SIGNALS); it is still reported on the decisionloop_lagormemoryalone, anything else needed a second signalmemory,gate_failuresand the newgate_slow_initsis sufficient alone (GATE_SUFFICIENT_ALONE); two other gate signals still corroborateSpawnGate.snapshot()reportsslow_inits: successful inits in the last 60 s that took ≥ 80 % of their owninitialize_timeout_secs; two of them firegate_slow_inits, so one slow but healthy server does notspawn_concurrency_initial, 60 s with none of its signals and no cut climbs it +1 per window, demand or not; growth past it still earns on initsgate_severe)The code is in
src/kiro_crew/adaptive(signals.py: the gate's signal set, thresholds and snapshot field;policy.py:_gate_step, its recovery and pause) andsrc/kiro_crew/mcp_gateway/admission.py(the slow-init count). The execution cap's rules are unchanged. Specs:docs/system-specs/modules/adaptive-concurrency.md,docs/system-specs/modules/mcp-gateway-daemon-lifecycle.md.The other two causes in #17206 (a slot held 10 s with no
initialize; a starting backend killed when its client gives up) are fixed separately in #17251.Backwards compatibility
Compatible: no config key, frame or public function is removed or renamed.
SpawnGateStatsand the gate snapshot only gain a field (an older daemon's snapshot reads as no slow inits), and the gate moves within the same configured bounds.Removes nothing: loop lag is still measured, classified and reported; only the gate track stops acting on it.
Tests
test/test_adaptive_policy.py: loop lag at 250 / 468 / 9000 ms moves neither cap and never pauses the gate; memory, gate failures and two slow inits each cut the gate alone; one slow init is no evidence; a cut gate climbs back to its fresh start with no demand and stops there; loop lag does not restart that recovery clock; snapshot parsing of the new fields, including an older or garbled snapshot; the gate's clear test ignores lag but not memory. Existing gate cases that used lag as the pressure now use memory.test/test_mcp_gateway_spawn_gate.py: an init counts as slow at 80 % of its own deadline (10 s and 30 s deadlines both), and the window ages out.test/test_adaptive_controller.py,test/test_subagent_memory_bounds_concurrency.py: updated to the gate holding under loop lag.Every test file touching the adaptive controller or the spawn gate (61 files, 5763 tests) passes locally; black, isort, flake8, mypy and docs-lint pass.
Manual verification
N/A — the policy is pure and driven by hand-built samples in the incident's shape (lag only, zero failures, idle host). After deploy,
resource_statusshould show the MCP spawn gate staying at its fresh-start value through dashboard loop lag.Related Issues
Closes #17206
Related: #10521, #17017, #17251.
Pattern harvest
Rule candidate: review-prompt
Pattern: an adaptive controller that moves one process's admission limit on another process's pressure signal, with no evidence from the work it limits.
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)