Repository navigation
fix(mcp-gateway): stop revoking a same-caller subscribe still in flight - #32317
Conversation
UX Review (Opus 5.5) — ⏭️ skippedRevision |
First Principles Review (Opus 5.5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Nothing blocks. The fix sends three release decisions through the existing What this change shipsInventory (6 items) — 6 justifiedIntent: on a server that tells callers apart, a session's resource subscription keeps getting updates when another stub of the same caller leaves while that subscribe is still in flight. FIX.
[FIRST-PRINCIPLES-REVIEWED] 8703ffc Description read[DESCRIPTION-READ] 9e55c45959c0daab97ad600967d89cfac508aab6cd07b4db8194096e4009c874 That sha256 names the PR description this verdict read. Recompute it with |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing blocks: three release sites that each had their own routed-only check now go through the existing Notes
[DESIGN-REVIEWED] 8703ffc Description read[DESCRIPTION-READ] 4e7efe7d04d14a55fa94e3b33cb6d4a93662837e3254bd54417fa64a5cc7102d That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
On a server with caller identity, a stub's detach, a warm-pool claim's eviction and the release sentinel's grant arm each released the caller's upstream subscription while another stub's subscribe for the same URI as that caller was still in flight. The server then granted that subscribe and dropped the caller's one subscription, leaving the new subscriber routed to nothing. All three now decide the release through _hold_identity_release, as a final unsubscribe already does: the in-flight subscribe inherits the release and its verdict settles it. The sentinel arm decides per caller, so a stub of another caller routed for the URI no longer keeps the caller's unwanted grant alive; a held release with nobody routed orphan-marks the URI so its updates are not charged as a hazard. Reported by javenciu in #21433.
22177fe to
8703ffc
Compare
|
self-added: yes
|
buluoray
left a comment
There was a problem hiding this comment.
Reviewed on 8703ffc: the three release sites now use the per-caller hold, so a same-caller subscribe in flight is not revoked while a different caller is still released; each new guard has a test that fails without it. Non-blocking: the timeout and backend-exit race paths were checked by reading, not by running.
…ht (#32317) On a server with caller identity, a stub's detach, a warm-pool claim's eviction and the release sentinel's grant arm each released the caller's upstream subscription while another stub's subscribe for the same URI as that caller was still in flight. The server then granted that subscribe and dropped the caller's one subscription, leaving the new subscriber routed to nothing. All three now decide the release through _hold_identity_release, as a final unsubscribe already does: the in-flight subscribe inherits the release and its verdict settles it. The sentinel arm decides per caller, so a stub of another caller routed for the URI no longer keeps the caller's unwanted grant alive; a held release with nobody routed orphan-marks the URI so its updates are not charged as a hazard. Reported by javenciu in #21433. (cherry picked from commit 21597a3)
Problem / Motivation
Goal: On an MCP server that tells callers apart, a session's resource subscription keeps receiving updates when another stub of the same caller leaves while that subscription request is still in flight.
On a server with caller identity the gateway forwards every
resources/subscribeunder its own caller, and the server keeps one subscription per URI and caller. When stub s1 holds a URI as caller A and stub s2's subscribe for the same URI as A is still unanswered, three events released A's subscription right beside s2's subscribe: s1 detaching, a warm-pool claim evicting s1's subscriptions, and the release sentinel's grant of a subscribe nobody was left to receive. The server then granted s2 and dropped A's one subscription, so s2 stayed routed for the URI and never heard another update. Reported by @javenciu in #21433, as the follow-up to their #18604.Why it matters
Any two stubs of one caller sharing a pooled identity-capable MCP server can hit it: two stubs claimed to one session, a warm-pool claim retargeting a stub, or a stub that leaves while its own subscribe is in flight. The loss is silent. The subscriber is told it is subscribed and simply stops receiving
notifications/resources/updated.Not a goal
_hold_identity_release), which fix(mcp-gateway): stop tracking a request its client cancelled #18604 added.What changed (motivation → approach → change)
A subscriber lost its updates because three release decisions looked only at stubs already routed for the URI and never at a same-caller subscribe still in flight. #18604 already has the rule that does:
_hold_identity_releaseholds a release while a stub of the caller is routed or a same-caller subscribe is in flight, which then owes the release (release_owed), and its verdict settles it. A final unsubscribe already goes through it.detach_stub(mcp_gateway/backend_stubs.py)_hold_identity_releaseevict_stub_subscriptions(mcp_gateway/backend_leases.py)_hold_identity_releasemcp_gateway/backend_routing.py)_hold_identity_releasewhen the server has caller identity and the subscribe a caller; otherwise unchangedmcp_gateway/backend_routing.py)release_owed)The sentinel arm now decides per caller, which changes one case: with only another caller's stub routed for the URI, it releases A's unwanted grant as A, as an owed release already did. A held release keeps the caller's subscription live upstream, so an update for it can arrive while no stub is routed for the URI, whichever caller's stub left last. That update is dropped without charging the server a hazard: the update arm reads the held release itself, and a sentinel grant held with no stub routed orphan-marks the URI, as the unsubscribe path does.
docs/architecture/mcp.mdnames every site the rule covers.Backwards compatibility
Compatible: nothing is rejected, renamed or removed; a release that went out beside a same-caller subscribe now waits for that subscribe's verdict, and every case with no such subscribe in flight decides exactly as before, except the sentinel's per-caller grant arm described above.
Removes nothing: the change reorders when the gateway sends its own
resources/unsubscribe; no record, key, path or public function goes away.Tests
test/test_mcp_gateway_backend_cancel.py,TestIdentityReleaseHold, one test per acceptance item of #21433:test_a_departing_holder_leaves_its_release_to_a_same_caller_subscribe(detach and evict, each granted and refused): s1 leaves while s2's subscribe as A is in flight; no release until s2's verdict, none on a grant, exactly one as A on a refusal.test_a_sentinel_grant_leaves_its_release_to_a_same_caller_subscribe(granted and refused): the sentinel's grant of s1's subscribe writes no release while s2's subscribe as A is in flight, orphan-marks the URI, and s2's verdict settles it.test_a_sentinel_grant_releases_as_its_caller_beside_another_callers_stub: pins the one behavior change; a stub of caller B routed for the URI does not keep A's unwanted grant.test_another_callers_departure_keeps_a_held_release_free_of_hazards(detach, evict, unsubscribe): A's release is held for s2's subscribe, then caller B's stub leaves and B's release is confirmed; an update for the URI records no hazard.test_identity_detach_releases_as_the_grant_caller,test_identity_detach_skips_release_for_a_shared_callerandtest_evict_stub_subscriptions_releases_as_grant_callerpass unchanged. Each new test drives the mock backend frame by frame, with no clock, sleep or port. Determinism: 20 runs of the hazard test at-n0, 10 runs at-n 4 --dist loadgroup --max-worker-restart=2of this file beside the two sibling lease test files, and one shuffled order (seed 31877) of the 44 tests in this file andtest/test_mcp_gateway_backend_coverage_caller_grants_eviction.py, all green.Proof: PROVEN test/test_mcp_gateway_backend_cancel.py::TestIdentityReleaseHold::test_another_callers_departure_keeps_a_held_release_free_of_hazards[detach], test/test_mcp_gateway_backend_cancel.py::TestIdentityReleaseHold::test_another_callers_departure_keeps_a_held_release_free_of_hazards[evict], test/test_mcp_gateway_backend_cancel.py::TestIdentityReleaseHold::test_another_callers_departure_keeps_a_held_release_free_of_hazards[unsubscribe], test/test_mcp_gateway_backend_cancel.py::TestIdentityReleaseHold::test_a_departing_holder_leaves_its_release_to_a_same_caller_subscribe[detach-granted], test/test_mcp_gateway_backend_cancel.py::TestIdentityReleaseHold::test_a_departing_holder_leaves_its_release_to_a_same_caller_subscribe[detach-refused] (head 8703ffc)
Manual verification
N/A — unit coverage sufficient: the race needs a server answering two subscribes in a chosen order, which the mock identity backend drives frame by frame.
Screenshots / video
N/A — backend-only change; no user-visible frontend surface.
Related Issues
Closes #21433
Builds on #18604 by @javenciu, which added
_hold_identity_releaseandrelease_owed.Sibling sweep
Swept: git grep -n "_release_upstream_subscriptions(" -- src/kiro_crew/mcp_gateway/
Sibling: src/kiro_crew/mcp_gateway/backend_stubs.py:235 -- changed here
Sibling: src/kiro_crew/mcp_gateway/backend_leases.py:245 -- changed here
Sibling: src/kiro_crew/mcp_gateway/backend_routing.py:525 -- changed here
The other release calls in those files are not affected:
backend_leases.py:658andbackend_routing.py:560already send an owed release through the hold;backend_leases.py:667andbackend_routing.py:566are the unsettled-verdict fail-closed releases left out under Not a goal;backend_routing.py:540,backend_leases.py:694andbackend_stubs.py:248serve a server without caller identity, or a subscribe with no caller, where one lease has no per-caller owner.Pattern harvest
Not generalizable: the rule lives in one helper, and every site where an identity release first falls due now calls it; a lint cannot tell a routed-only check from a correct one without the lease model.
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)Contribution License Agreement
Contributed under the project's Apache-2.0 license.