Skip to content

fix: move the spawn gate on its own evidence, not gateway loop lag - #17231

Merged
chenmingwei23 merged 1 commit into
mainfrom
fix/spawn-gate-own-signals
Oct 6, 2026
Merged

chenmingwei23 merged 1 commit into
mainfrom
fix/spawn-gate-own-signals

Conversation

@bolichen97

@bolichen97 bolichen97 commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: New chats get their MCP tools on an idle host instead of failing the 90 s readiness timeout.

On one host (2026-10-05) gatewayd's spawn gate sat at 1 of 8 with 0 spawn failures out of 621, load 0.09 per core and 40 GB free. It was cut to 1 49 times that day, each time on corroborated pressure: loop_lag from the dashboard gateway process. With the gate at 1, backends start one at a time and each holds the slot for up to 10 s (95 of 621 permits held the whole initialize_timeout_secs with no initialize). kirocrew-core / kirocrew-work queue behind external MCPs, the client detaches its stub after ~30 s, gatewayd kills the still-starting backend (recycled (killed): last stub detached with in-flight work) and the retry goes to the back of the queue, until the chat times out. See #17206.

Why it matters

Every new chat on an affected host can come up with no MCP tools, on a machine that is doing nothing. One dashboard event-loop hiccup is enough to start it, and the gate does not recover on its own.

Not a goal

What changed (motivation → approach → change)

The spawn gate stopped reading the dashboard gateway's loop lag and now moves on its own evidence; a cut gate climbs back by itself.

Before (main after #17017) Now
loop_lag alone cut the gate; lag over 2 s paused it loop_lag never moves the gate (GATE_IGNORED_SIGNALS); it is still reported on the decision
Gate cut on loop_lag or memory alone, anything else needed a second signal Each of memory, gate_failures and the new gate_slow_inits is sufficient alone (GATE_SUFFICIENT_ALONE); two other gate signals still corroborate
No signal for how slow backend starts are SpawnGate.snapshot() reports slow_inits: successful inits in the last 60 s that took ≥ 80 % of their own initialize_timeout_secs; two of them fire gate_slow_inits, so one slow but healthy server does not
A cut gate grew back only on 20 clean inits with demand at the cap Below spawn_concurrency_initial, 60 s with none of its signals and no cut climbs it +1 per window, demand or not; growth past it still earns on inits
Pause on memory or lag Pause only on critical memory (gate_severe)

The code is in src/kiro_crew/adaptive (signals.py: the gate's signal set, thresholds and snapshot field; policy.py: _gate_step, its recovery and pause) and src/kiro_crew/mcp_gateway/admission.py (the slow-init count). The execution cap's rules are unchanged. Specs: docs/system-specs/modules/adaptive-concurrency.md, docs/system-specs/modules/mcp-gateway-daemon-lifecycle.md.

The other two causes in #17206 (a slot held 10 s with no initialize; a starting backend killed when its client gives up) are fixed separately in #17251.

Backwards compatibility

Compatible: no config key, frame or public function is removed or renamed. SpawnGateStats and the gate snapshot only gain a field (an older daemon's snapshot reads as no slow inits), and the gate moves within the same configured bounds.

Removes nothing: loop lag is still measured, classified and reported; only the gate track stops acting on it.

Tests

  • test/test_adaptive_policy.py: loop lag at 250 / 468 / 9000 ms moves neither cap and never pauses the gate; memory, gate failures and two slow inits each cut the gate alone; one slow init is no evidence; a cut gate climbs back to its fresh start with no demand and stops there; loop lag does not restart that recovery clock; snapshot parsing of the new fields, including an older or garbled snapshot; the gate's clear test ignores lag but not memory. Existing gate cases that used lag as the pressure now use memory.
  • test/test_mcp_gateway_spawn_gate.py: an init counts as slow at 80 % of its own deadline (10 s and 30 s deadlines both), and the window ages out.
  • test/test_adaptive_controller.py, test/test_subagent_memory_bounds_concurrency.py: updated to the gate holding under loop lag.

Every test file touching the adaptive controller or the spawn gate (61 files, 5763 tests) passes locally; black, isort, flake8, mypy and docs-lint pass.

Manual verification

N/A — the policy is pure and driven by hand-built samples in the incident's shape (lag only, zero failures, idle host). After deploy, resource_status should show the MCP spawn gate staying at its fresh-start value through dashboard loop lag.

Related Issues

Closes #17206

Related: #10521, #17017, #17251.

Pattern harvest

Rule candidate: review-prompt
Pattern: an adaptive controller that moves one process's admission limit on another process's pressure signal, with no evidence from the work it limits.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@bolichen97
bolichen97 requested a review from a team as a code owner October 6, 2026 04:32
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 6, 2026
@github-actions

github-actions Bot commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — 🟡 CONCERNS

Premise-level review of d4519cfab304710d69d8dde417d25ed7e26119ec — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: CONCERNS

Two riders add new ways to cut the gate, but the incident had zero failures and no slow inits; ignoring lag plus idle recovery already fixes #17206.

Not justified as shipped

  • Item 3 — rides along, inherited: the only stated support is the before-state "No signal for how slow backend starts are". No issue, failing test or reproduction shows a slow successful init that should have cut the gate.
    Subtraction: delete SIGNAL_GATE_SLOW_INITS, Thresholds.gate_slow_inits, SLOW_INIT_FRACTION/SLOW_INIT_WINDOW_SECS, SpawnGate._slow_inits/slow_init_count, and the slow_inits snapshot/SpawnGateStats field.
    Clears when: a reported case where slow successful inits overloaded the host and the gate held, or the deletion is made.
  • Item 2 — rides along, inherited: GATE_SUFFICIENT_ALONE now includes SIGNAL_GATE_FAILURES, so two failures in 60 s alone halve the gate. Before this change they needed a second signal. adaptive: dashboard loop_lag alone pins gatewayd spawn gate at 1, MCP readiness times out on an idle host #17206 had "0 spawn failures out of 621", so this rider is unrelated to the defect.
    Subtraction: shrink GATE_SUFFICIENT_ALONE to {SIGNAL_MEMORY}.
    Clears when: an issue shows uncorroborated gate failures that should have cut the gate, or the shrink is made.

What this change ships

Inventory (6 items) — 4 justified

Intent: let new chats on an idle host get their MCP tools instead of timing out. This is a FIX.

  1. Dashboard loop lag no longer cuts or pauses the MCP spawn gate (still reported) — justified
  2. Two gate init failures alone now cut the gate — rides along
  3. New "slow inits" count in the daemon snapshot, which can cut the gate alone — rides along
  4. A cut gate climbs back to its fresh-start value without demand — justified
  5. Gate pauses only on critical memory — justified
  6. Gate's increase test drops the lag hysteresis band — justified

[FIRST-PRINCIPLES-REVIEWED] d4519cf

Description read

[DESCRIPTION-READ] bae931ef4e9d86b75c13b216c6b6174f4ab558cc874d08e4e6dec3acc68bd796

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of d4519cfab304710d69d8dde417d25ed7e26119ec — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Check that slow_inits timing does not count the client's own late initialize against the backend before #17251 lands.

The timer starts at permit.granted_at and stops when the backend reports ready. That span includes any time the client waits before sending initialize, which is one of the #17206 causes #17251 fixes. Until #17251 merges, a late client could make inits look slow, fire gate_slow_inits, and cut the gate on a quiet host again. Idle recovery would bring it back up after 60 s.

[DESIGN-REVIEWED] d4519cf

Description read

[DESCRIPTION-READ] 3515b0c86114d8548009beeb93675d321af3ba329bd5b6931c7a4118a28fe232

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 6, 2026
@github-actions

github-actions Bot commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed d4519cfab304710d69d8dde417d25ed7e26119ec — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] d4519cf

Verdict parsed from the review's SHA-scoped output markers for commit d4519cfab304710d69d8dde417d25ed7e26119ec.

False positive or not applicable? A repository writer can comment:
/ai-review override fable d4519cfab304710d69d8dde417d25ed7e26119ec: <one-sentence reason>

@bolichen97
bolichen97 enabled auto-merge (squash) October 6, 2026 04:42
@github-actions

github-actions Bot commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of d4519cfab304710d69d8dde417d25ed7e26119ec and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] d4519cf

False positive or not applicable? A repository writer can comment:
/ai-review override gpt d4519cfab304710d69d8dde417d25ed7e26119ec: <one-sentence reason>

@bolichen97
bolichen97 marked this pull request as draft October 6, 2026 05:53
auto-merge was automatically disabled October 6, 2026 05:53

Pull request was converted to draft

@bolichen97

Copy link
Copy Markdown
Contributor Author

Holding this as a draft.

#17017 rewrites src/kiro_crew/adaptive/policy.py and moves the spawn gate in the opposite direction: there loop lag and memory alone still cut the gate, while this PR cuts it only on gatewayd's own spawn evidence. Only one of those rules can land, so the gate-signal change waits for #17017 to merge and will then be rebased on top of it, changing only the gate track.

The two parts that do not touch the policy (free a slot after 2 s with no initialize; keep a starting backend when its client abandons the handshake) are split out to their own PR, Part of #17206.

@github-actions github-actions Bot added the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Oct 6, 2026
The spawn gate admits forks inside the MCP daemon, but one loop-lag
sample from the dashboard gateway was enough to cut it, and severe lag
paused it. On an idle host with zero spawn failures it sat at 1 all
day, backends started one at a time and new chats missed the MCP
readiness deadline.

The gate track now ignores loop lag. It cuts on critical memory, on
gate init failures, or on two slow inits (each at 80% of its own
initialize deadline, within 60 s), each sufficient alone, and pauses only on
critical memory. A cut gate climbs back one step per clean window to
its fresh-start value without needing queued demand. The exec track is
unchanged.

Closes #17206
@bolichen97 bolichen97 changed the title fix: move the spawn gate on gatewayd's own spawn evidence fix: move the spawn gate on its own evidence, not gateway loop lag Oct 6, 2026
@bolichen97
bolichen97 force-pushed the fix/spawn-gate-own-signals branch from d0b346f to d4519cf Compare October 6, 2026 06:33
@bolichen97
bolichen97 marked this pull request as ready for review October 6, 2026 06:34
@bolichen97

Copy link
Copy Markdown
Contributor Author

Rebased onto main now that #17017 has landed, and narrowed to the gate track only: the gate stops acting on the gateway loop lag and moves on its own evidence (critical memory, gate init failures, slow inits relative to each init's own deadline), with idle recovery. The exec track #17017 shaped is untouched. The teardown fix is in #17251; the 2 s early slot release is dropped.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention merge conflict Branch has merge conflicts with its base — author must resolve before merge readiness: checking Automated validation is still running labels Oct 6, 2026
@bolichen97

Copy link
Copy Markdown
Contributor Author
  • Item 3 - gate_slow_inits rides along without a reported case (span=78bd11d81714)

rebutted: it is the replacement for the signal this PR removes, not a rider.
Loop lag was the gate's only congestion proxy; the frozen Not-a-goal forbids loosening SpawnGate (#10521).
Without a latency signal the gate would only react once forks fail outright, i.e. after the storm it exists to bound.
#17206's own proposal names slow inits as the gate's evidence. It is bounded: 2 inits at 80% of their own deadline in 60 s.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • Item 2 - gate_failures sufficient alone (span=64b13f9a528c)

rebutted: before, loop lag alone could cut the gate while failing forks needed a second signal.
With lag removed, keeping that rule means a host whose forks are failing (EAGAIN/ENOMEM, the #10521 storm) waits for a coincident signal.
The gate's own failures are its most direct evidence; threshold stays at 2 in a 60 s window, so one failure does not cut.
Not-a-goal: no loosening of SpawnGate's protection; this keeps it equivalent after lag is gone.

@bolichen97
bolichen97 enabled auto-merge (squash) October 6, 2026 07:21
@chenmingwei23
chenmingwei23 merged commit 9f05628 into main Oct 6, 2026
103 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

adaptive: dashboard loop_lag alone pins gatewayd spawn gate at 1, MCP readiness times out on an idle host

2 participants