Skip to content

fix(mcp-gateway): stop revoking a same-caller subscribe still in flight - #32317

Merged
bolichen97 merged 1 commit into
mainfrom
fix/21433-identity-release-in-flight
Nov 7, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
fix/21433-identity-release-in-flight

Conversation

@bolichen97

@bolichen97 bolichen97 commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: On an MCP server that tells callers apart, a session's resource subscription keeps receiving updates when another stub of the same caller leaves while that subscription request is still in flight.

On a server with caller identity the gateway forwards every resources/subscribe under its own caller, and the server keeps one subscription per URI and caller. When stub s1 holds a URI as caller A and stub s2's subscribe for the same URI as A is still unanswered, three events released A's subscription right beside s2's subscribe: s1 detaching, a warm-pool claim evicting s1's subscriptions, and the release sentinel's grant of a subscribe nobody was left to receive. The server then granted s2 and dropped A's one subscription, so s2 stayed routed for the URI and never heard another update. Reported by @javenciu in #21433, as the follow-up to their #18604.

Why it matters

Any two stubs of one caller sharing a pooled identity-capable MCP server can hit it: two stubs claimed to one session, a warm-pool claim retargeting a stub, or a stub that leaves while its own subscribe is in flight. The loss is silent. The subscriber is told it is subscribed and simply stops receiving notifications/resources/updated.

Not a goal

What changed (motivation → approach → change)

A subscriber lost its updates because three release decisions looked only at stubs already routed for the URI and never at a same-caller subscribe still in flight. #18604 already has the rule that does: _hold_identity_release holds a release while a stub of the caller is routed or a same-caller subscribe is in flight, which then owes the release (release_owed), and its verdict settles it. A final unsubscribe already goes through it.

Release site Before After
detach_stub (mcp_gateway/backend_stubs.py) held only for a routed same-caller stub _hold_identity_release
evict_stub_subscriptions (mcp_gateway/backend_leases.py) held only for a routed same-caller stub _hold_identity_release
sentinel grant, no release owed (mcp_gateway/backend_routing.py) held while any caller's stub was routed _hold_identity_release when the server has caller identity and the subscribe a caller; otherwise unchanged
update for a URI no stub is routed for (mcp_gateway/backend_routing.py) no hazard only when the URI is orphan-marked no hazard either while a same-caller subscribe in flight owes a held release (release_owed)

The sentinel arm now decides per caller, which changes one case: with only another caller's stub routed for the URI, it releases A's unwanted grant as A, as an owed release already did. A held release keeps the caller's subscription live upstream, so an update for it can arrive while no stub is routed for the URI, whichever caller's stub left last. That update is dropped without charging the server a hazard: the update arm reads the held release itself, and a sentinel grant held with no stub routed orphan-marks the URI, as the unsubscribe path does. docs/architecture/mcp.md names every site the rule covers.

Backwards compatibility

Compatible: nothing is rejected, renamed or removed; a release that went out beside a same-caller subscribe now waits for that subscribe's verdict, and every case with no such subscribe in flight decides exactly as before, except the sentinel's per-caller grant arm described above.

Removes nothing: the change reorders when the gateway sends its own resources/unsubscribe; no record, key, path or public function goes away.

Tests

test/test_mcp_gateway_backend_cancel.py, TestIdentityReleaseHold, one test per acceptance item of #21433:

  • test_a_departing_holder_leaves_its_release_to_a_same_caller_subscribe (detach and evict, each granted and refused): s1 leaves while s2's subscribe as A is in flight; no release until s2's verdict, none on a grant, exactly one as A on a refusal.
  • test_a_sentinel_grant_leaves_its_release_to_a_same_caller_subscribe (granted and refused): the sentinel's grant of s1's subscribe writes no release while s2's subscribe as A is in flight, orphan-marks the URI, and s2's verdict settles it.
  • test_a_sentinel_grant_releases_as_its_caller_beside_another_callers_stub: pins the one behavior change; a stub of caller B routed for the URI does not keep A's unwanted grant.
  • test_another_callers_departure_keeps_a_held_release_free_of_hazards (detach, evict, unsubscribe): A's release is held for s2's subscribe, then caller B's stub leaves and B's release is confirmed; an update for the URI records no hazard.

test_identity_detach_releases_as_the_grant_caller, test_identity_detach_skips_release_for_a_shared_caller and test_evict_stub_subscriptions_releases_as_grant_caller pass unchanged. Each new test drives the mock backend frame by frame, with no clock, sleep or port. Determinism: 20 runs of the hazard test at -n0, 10 runs at -n 4 --dist loadgroup --max-worker-restart=2 of this file beside the two sibling lease test files, and one shuffled order (seed 31877) of the 44 tests in this file and test/test_mcp_gateway_backend_coverage_caller_grants_eviction.py, all green.

Proof: PROVEN test/test_mcp_gateway_backend_cancel.py::TestIdentityReleaseHold::test_another_callers_departure_keeps_a_held_release_free_of_hazards[detach], test/test_mcp_gateway_backend_cancel.py::TestIdentityReleaseHold::test_another_callers_departure_keeps_a_held_release_free_of_hazards[evict], test/test_mcp_gateway_backend_cancel.py::TestIdentityReleaseHold::test_another_callers_departure_keeps_a_held_release_free_of_hazards[unsubscribe], test/test_mcp_gateway_backend_cancel.py::TestIdentityReleaseHold::test_a_departing_holder_leaves_its_release_to_a_same_caller_subscribe[detach-granted], test/test_mcp_gateway_backend_cancel.py::TestIdentityReleaseHold::test_a_departing_holder_leaves_its_release_to_a_same_caller_subscribe[detach-refused] (head 8703ffc)

Manual verification

N/A — unit coverage sufficient: the race needs a server answering two subscribes in a chosen order, which the mock identity backend drives frame by frame.

Screenshots / video

N/A — backend-only change; no user-visible frontend surface.

Related Issues

Closes #21433

Builds on #18604 by @javenciu, which added _hold_identity_release and release_owed.

Sibling sweep

Swept: git grep -n "_release_upstream_subscriptions(" -- src/kiro_crew/mcp_gateway/
Sibling: src/kiro_crew/mcp_gateway/backend_stubs.py:235 -- changed here
Sibling: src/kiro_crew/mcp_gateway/backend_leases.py:245 -- changed here
Sibling: src/kiro_crew/mcp_gateway/backend_routing.py:525 -- changed here

The other release calls in those files are not affected: backend_leases.py:658 and backend_routing.py:560 already send an owed release through the hold; backend_leases.py:667 and backend_routing.py:566 are the unsettled-verdict fail-closed releases left out under Not a goal; backend_routing.py:540, backend_leases.py:694 and backend_stubs.py:248 serve a server without caller identity, or a subscribe with no caller, where one lease has no per-caller owner.

Pattern harvest

Not generalizable: the rule lives in one helper, and every site where an identity release first falls due now calls it; a lint cannot tell a routed-only check from a correct one without the lease model.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributed under the project's Apache-2.0 license.

@bolichen97
bolichen97 requested review from a team and KiroCrewCaptain as code owners November 7, 2026 12:31
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 7, 2026
@github-actions

github-actions Bot commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 8703ffccd5d88c5e0d60155849aee6053b99508b touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of 8703ffccd5d88c5e0d60155849aee6053b99508b — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Nothing blocks. The fix sends three release decisions through the existing _hold_identity_release and deletes two copies of an inline routed-only check. Issue #21433 is the reported defect, and new tests pin each case.

What this change ships

Inventory (6 items) — 6 justified

Intent: on a server that tells callers apart, a session's resource subscription keeps getting updates when another stub of the same caller leaves while that subscribe is still in flight. FIX.

  1. When a stub detaches, it no longer drops caller A's upstream subscription while A's subscribe is still in flight; the subscribe's verdict settles the release. — justified
  2. When a warm-pool claim evicts a stub's subscriptions, the same hold applies. — justified
  3. When the release sentinel's unwanted grant arrives during a same-caller subscribe, the release is held and the URI is orphan-marked. — justified
  4. The sentinel now releases A's unwanted grant as A even while another caller's stub is routed for the URI. — justified
  5. An update for a URI whose release is held and has no routed stub is dropped without recording a hazard against the server. — justified
  6. docs/architecture/mcp.md and the docstrings now name every site the hold rule covers. — justified

[FIRST-PRINCIPLES-REVIEWED] 8703ffc

Description read

[DESCRIPTION-READ] 9e55c45959c0daab97ad600967d89cfac508aab6cd07b4db8194096e4009c874

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 8703ffccd5d88c5e0d60155849aee6053b99508b — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: three release sites that each had their own routed-only check now go through the existing _hold_identity_release rule, and the one changed behaviour (the sentinel releases per caller) is described and tested.

Notes

  • In evict_stub_subscriptions, check that the evicted stub's own subscribes still in flight are removed from _pending_requests before _hold_identity_release runs. If one is still there, it can take release_owed for itself, and the release then depends on how that retracted subscribe settles.

[DESIGN-REVIEWED] 8703ffc

Description read

[DESCRIPTION-READ] 4e7efe7d04d14a55fa94e3b33cb6d4a93662837e3254bd54417fa64a5cc7102d

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 8703ffccd5d88c5e0d60155849aee6053b99508b — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 8703ffc

Verdict parsed from the review's SHA-scoped output markers for commit 8703ffccd5d88c5e0d60155849aee6053b99508b.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 8703ffccd5d88c5e0d60155849aee6053b99508b false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 8703ffccd5d88c5e0d60155849aee6053b99508b and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 8703ffc

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 8703ffccd5d88c5e0d60155849aee6053b99508b false-positive: <reason, 20 to 500 characters>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 7, 2026
@bolichen97
bolichen97 enabled auto-merge (squash) November 7, 2026 12:44
On a server with caller identity, a stub's detach, a warm-pool claim's
eviction and the release sentinel's grant arm each released the
caller's upstream subscription while another stub's subscribe for the
same URI as that caller was still in flight. The server then granted
that subscribe and dropped the caller's one subscription, leaving the
new subscriber routed to nothing.

All three now decide the release through _hold_identity_release, as a
final unsubscribe already does: the in-flight subscribe inherits the
release and its verdict settles it. The sentinel arm decides per
caller, so a stub of another caller routed for the URI no longer keeps
the caller's unwanted grant alive; a held release with nobody routed
orphan-marks the URI so its updates are not charged as a hazard.

Reported by javenciu in #21433.
@bolichen97
bolichen97 force-pushed the fix/21433-identity-release-in-flight branch from 22177fe to 8703ffc Compare November 7, 2026 16:25
@bolichen97

bolichen97 commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor Author

self-added: yes
mechanism: the resources/updated arm treats an in-flight subscribe owing a held release (release_owed) as an orphan mark

  • Another caller's departure leaves a held subscription unprotected span=c969835df95d

fixed in 8703ffc: an update for a URI no stub is routed for is dropped without a hazard while any in-flight subscribe for that URI owes a held release (release_owed), not only while the URI is orphan-marked.
The check reads the hold itself at drop time, so it holds however routing empties or the URI-keyed mark clears while a release is held: detach, eviction, a forwarded final unsubscribe, or another caller's confirmed release.
Pinned by test_another_callers_departure_keeps_a_held_release_free_of_hazards (detach, evict, unsubscribe): red on all three without the change, green with it.
[operator: bolichen97]

@bolichen97

Copy link
Copy Markdown
Contributor Author

Rebased: main moved in docs/architecture/mcp.md, docs/architecture/context-management.md, src/kiro_crew/acp/harness/kas.py, test/conftest.py since this head went green (bab2387 -> 759bb96); CI re-running.

[operator: bolichen97]

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Nov 7, 2026

@buluoray buluoray left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed on 8703ffc: the three release sites now use the per-caller hold, so a same-caller subscribe in flight is not revoked while a different caller is still released; each new guard has a test that fails without it. Non-blocking: the timeout and backend-exit race paths were checked by reading, not by running.

@bolichen97
bolichen97 merged commit 21597a3 into main Nov 7, 2026
99 checks passed
@bolichen97
bolichen97 deleted the fix/21433-identity-release-in-flight branch November 7, 2026 18:48
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Nov 7, 2026
buluoray pushed a commit that referenced this pull request Nov 7, 2026
…ht (#32317)

On a server with caller identity, a stub's detach, a warm-pool claim's
eviction and the release sentinel's grant arm each released the
caller's upstream subscription while another stub's subscribe for the
same URI as that caller was still in flight. The server then granted
that subscribe and dropped the caller's one subscription, leaving the
new subscriber routed to nothing.

All three now decide the release through _hold_identity_release, as a
final unsubscribe already does: the in-flight subscribe inherits the
release and its verdict settles it. The sentinel arm decides per
caller, so a stub of another caller routed for the URI no longer keeps
the caller's unwanted grant alive; a held release with nobody routed
orphan-marks the URI so its updates are not charged as a hazard.

Reported by javenciu in #21433.

(cherry picked from commit 21597a3)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

mcp-gateway: route detach, evict and the sentinel grant arm through _hold_identity_release

2 participants