Skip to content

fix(mcp-gateway): stop tracking a request its client cancelled - #18604

Merged
javenciu merged 2 commits into
kirodotdev:mainfrom
javenciu:fix/mcp-gateway-drops-cancelled-pending
Oct 29, 2026
Merged

javenciu merged 2 commits into
kirodotdev:mainfrom
javenciu:fix/mcp-gateway-drops-cancelled-pending

Conversation

@javenciu

@javenciu javenciu commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A request its client cancels stops being tracked as in flight, so it cannot age into the heartbeat's wedge ceiling and recycle a healthy shared MCP backend. For a tool call that means the entry is retired while the frames still tied to it (a progress notification, a sampling request) keep reaching the stub that made it. For a resources/subscribe or resources/unsubscribe that holds a URI's lease it means the request is handed off, so the server's answer still settles the lease for the stubs waiting on it.

Backend.forward_from_stub handles a stub's notifications/cancelled by remapping params.requestId to the gateway-scoped id and forwarding it. It keeps the request's _pending_requests entry. A server that honours an MCP cancellation sends no response, so the entry is never settled. _heartbeat_once reads the oldest pending age, and at HARD_WEDGE_CEILING_SECS (SUBAGENT_TIMEOUT_SECS + 300, 11,100 s) it declares the backend wedged whatever its pings say. _broadcast_backend_gone then fails the in-flight calls of every session on that backend, and the backend is recycled.

The stub has the same gap. run_bridge adds each request id to _outstanding_ids and removes it only on a response, so a cancelled id keeps the liveness monitor pinging, and a later bridge drop answers the cancelled call with -32603.

Why it matters

Cancelling a call is ordinary: stopping a turn sends one. About three hours later, while the cancelling session stays attached, the heartbeat recycles a backend that still answers pings. On kirocrew-core that fails every co-pooled session's running wait or spawn call.

Not a goal

  • Cancels the daemon sends itself (cancel_in_flight_for_stub, from disconnect, abort and app-call teardown). Their entries are what has_recyclable_in_flight reads to recycle a backend whose last stub left, because a cancel is best-effort, and each is cleared by the detach_stub that follows or by the stub's own disconnect.
  • The reconnect path's "Retry it." answer to a call the dead daemon may already have forwarded. That is a question about repeated side effects, not this cause.
  • Late answers that are oversize or do not parse. They find no pending entry and are dropped as they are on main: the unparseable path logs at debug, the oversize path keeps its frame-size warning.
  • Other responses to unknown ids keep their warning, including one for a cancelled id older than the last _CANCELLED_FIDS_KEPT cancels.

What changed (motivation → approach → change)

  • Motivation: since fix(mcp): park waiting tools off the worker so a wait stalls nobody #18841 a cancelled request is marked (_PendingRequest.cancelled) and the hard ceiling skips it, but the mark leaves out a resources/subscribe or resources/unsubscribe, whose verdict settles the shared lease tables. Its cancel is still forwarded, and its entry stays pending, unmarked. A server that honours the cancel sends no verdict, so the URI stays in _lease_awaiting_grant (a first subscribe) or _lease_awaiting_release (a final unsubscribe), every stub parked behind it waits for the backend's lifetime, and the entry ages into the ceiling.
  • src/kiro_crew/mcp_gateway/backend_stubs.py, the cancel branch of forward_from_stub: for a subscribe or unsubscribe (an entry with resource_uri) the cancel is not forwarded. The entry stays owned by the stub that sent it, so the server answers it and the verdict settles the lease through the arms main already uses when a server answers a request it was asked to cancel: the URI leaves the awaiting set, parked stubs get their verdict, and a confirmed unsubscribe stops routing the stub. Every other cancelled request is marked and forwarded exactly as on main.
  • src/kiro_crew/mcp_gateway/backend_leases.py, identity-capable server, a stub's final unsubscribe: the existing sharer guard (another routed stub of the same caller: settle locally, forward nothing) now also covers a same-caller subscribe still in flight. Forwarding the unsubscribe beside it would let the server grant that subscribe and then drop the caller's one subscription, leaving the new subscriber routed to nothing. The stub is unrouted and answered locally, and the release is left to that subscribe's verdict (_PendingRequest.release_owed, backend_wire.py). One rule, _hold_identity_release (backend_leases.py, bound on Backend), decides both that step and every later settlement of an owed release: it is sent only when no stub of the caller is routed for the URI and no other same-caller subscribe for it is in flight; a routed stub absorbs it (its own departure releases, as on main), and an in-flight subscribe inherits it (release_owed moves to that pending). Both settlement arms call it: the stub-owned verdict arm (_on_resource_subscription_response) and the release sentinel's orphan arm in _settle (backend_routing.py), on a grant as well as on a refusal. So with several same-caller subscribes in flight the outcome does not depend on which answers first: no release while one is granted or still in flight, exactly one, as the caller, when none is.
  • src/kiro_crew/mcp_gateway/stub_bridge.py, run_bridge: a notifications/cancelled from kiro-cli discards its requestId from _outstanding_ids.
  • Unchanged from fix(mcp): park waiting tools off the worker so a wait stalls nobody #18841: a cancelled tool call stays pending and marked. This PR does not retire it. The hard ceiling skips it, while the two-condition rule, notification attribution and the last-stub recycle (has_recyclable_in_flight) still count it.
  • Unchanged from main: detach, evict, retract and replay, and every release the gateway sends on its own. No subscribe is refused that main forwarded.
  • Rejected: handing a cancelled lease request to a parked rider or the release sentinel, as an earlier revision of this PR did. Its late grant then needed a gateway release, which on an identity server can land after a same-caller grant. Covering that took a release-ordering rule and a local "retry" refusal that no client is shown to retry. Keeping the request with its stub needs neither.
  • Rejected: raising the ceiling, which only moves the date, and exempting a backend with fresh pings from the ceiling, which removes the guard for a call that really is stuck.

Backwards compatibility

Compatible: requests that are not cancelled are tracked, answered and recycled exactly as on main, and every cancelled request other than a subscribe or unsubscribe is marked exactly as on main. A cancelled subscribe or unsubscribe is no longer sent its cancel, so the server answers it and the stub that sent it receives the verdict. On main a server that ignores a cancel produces the same outcome.

Accepted: a granted subscribe whose client cancelled it stays with that stub. The stub is routed and the upstream lease is held until the stub unsubscribes or detaches, both of which release as on main, and the client may receive notifications/resources/updated for it. MCP makes cancellation best-effort: a receiver may ignore it and the requester must be ready for the response anyway, so this is a state every client already handles, and on main it is the outcome with any server that ignores or outruns the cancel. Releasing the grant from the gateway instead would need a gateway-originated release as the caller beside whatever same-caller subscribes are in flight, the ordering this PR's earlier revisions could not make safe; keeping the grant with the stub creates no release at all.

On an identity-capable server, a stub's final unsubscribe sent while a same-caller subscribe for that URI is in flight is answered locally and its release waits for that subscribe's verdict, where main forwarded it at once. The cancel change is what makes this necessary: on main the cancel of such an unsubscribe is forwarded, and a server that honours it never applies the unsubscribe, so the in-flight subscribe keeps its grant (the stub is left pending instead, which is this PR's bug). This PR lands every cancelled unsubscribe, so without the rule it would revoke the sibling's grant on every such server. Main has the same race only on a server that ignores the cancel, or when nothing was cancelled. The owning spec, docs/architecture/mcp.md, now states this rule after its cancel paragraph.
No config key, API, frame shape or public function changes.

Tests

In short: eleven new cases fail on main 1b96d89 and pass here. Four of them also fail on this PR's previous head 1731193, which decided a deferred release by the first same-caller subscribe found; two fail on the head before it, e8f4897. With the stub and bindings tests and main's backend coverage file, 282 passed in each of 3 of 3 runs, RC=0.

Red on main (this branch's test/test_mcp_gateway_backend_cancel.py copied into a detached checkout of main 1b96d89, imported from that checkout's own src), pytest test/test_mcp_gateway_backend_cancel.py -p no:cacheprovider -q:

================== 11 failed, 2 passed, 33 warnings in 8.15s ===================

In test/test_mcp_gateway_backend_cancel.py (TestForwardFromStubCancel and TestIdentityReleaseHold), failing on main:

  • test_a_cancelled_subscribe_does_not_age_a_healthy_backend_into_the_ceiling: s1 takes a URI's lease, s2 parks behind it, s1 cancels. No cancel is forwarded, the server answers, s2 is granted, nothing stays pending, and past the hard ceiling with fresh pings the backend reads alive.
  • test_a_cancelled_final_unsubscribe_settles_the_release_under_its_stub: the cancel of a final unsubscribe is not forwarded; its success clears routing and _lease_awaiting_release.
  • test_an_identity_cancelled_subscribe_is_settled_by_its_own_verdict: on an identity server the grant routes the stub as the caller that took it, and no release is sent.
  • test_a_last_holder_unsubscribe_waits_for_a_same_caller_subscribe[granted|refused]: s1 holds a URI as caller A, s2's subscribe as A is in flight, and s1 unsubscribes and cancels. No release goes out beside s2's subscribe. A grant routes s2 with no release, and a refusal sends exactly one, carrying A's caller block.
  • test_a_release_left_to_one_of_two_same_caller_subscribes_spares_the_other (3 cases): s2 and s3 both have a subscribe as A in flight when s1, the last holder, unsubscribes. s3 granted then s2 refused, and s2 refused then s3 granted, each end with s3 routed as A and no release; both refused ends with exactly one release, as A. The first two fail on the previous head 1731193, which released on s2's refusal in both orders.
  • test_a_release_left_to_a_detached_stubs_subscribe_spares_a_routed_sibling[granted|refused]: the subscribe carrying the release belongs to s2, which detaches before the server answers; s3 is granted as A meanwhile. s2's verdict settles under the release sentinel and releases nothing. The refused case fails on the previous head.
  • test_a_retracted_subscribe_carrying_the_release_settles_it_beside_another_caller: s1 holds a URI as A and re-subscribes it, sb holds it as B, and s1 unsubscribes while its re-subscribe is in flight. The retracted subscribe inherits the release, and its grant sends exactly one release, as A, with sb still routed. Fails on the previous head, which released nothing because the URI was still routed for B.

Passing on main as well: test_a_cancelled_subscribe_the_server_never_answers_still_ages_out (an unanswered subscribe still trips the ceiling, and the recycle answers the parked rider) and test_a_hung_backend_whose_only_call_was_cancelled_is_recycled (stale pings still recycle a backend whose only call was cancelled, #18841's mark through a real cancel).

In test/test_stub_bridge_liveness.py, failing on main:

  • test_a_cancelled_request_is_no_longer_outstanding: through the real run_bridge, the cancelled id is dropped and an uncancelled one stays outstanding.

test/test_backend_bindings.py lists the one new Backend member, _hold_identity_release, in the frozen inventory.

Green: pytest test/test_mcp_gateway_backend_cancel.py test/test_stub_bridge_liveness.py test/test_backend_bindings.py test/test_mcp_gateway_backend_coverage.py -p no:cacheprovider -q -n 8, 3 of 3 runs, 282 passed each (13 + 10 + 172 + 87). flake8, isort and the black ratchet pass on the touched files.

Manual verification

None beyond the tests: the backend tests drive the real forward_from_stub, _route_backend_line and _heartbeat_once over a mock process, and the stub test drives the real run_bridge.

Screenshots / video

N/A: no file under website/ changes.

Related Issues

Builds on #18841, which marks a cancelled request so the hard ceiling skips it; this PR covers the subscribe and unsubscribe that the mark leaves out. No issue found: searched PRs and issues for "notifications/cancelled", "cancelled request pending", "hard wedge ceiling", "wedge cancelled" and "_pending_requests".

Follow-up: #21433 routes detach, evict and the sentinel's grant arm through _hold_identity_release, so a same-caller subscribe still in flight is counted there.

Pattern harvest

Rule candidate: review-prompt: before a change drops a request that ended without a response (a cancel, a timeout) from the table that tracks it, list every reader of that table; a liveness ceiling, notification attribution and a last-consumer recycle can each need the entry for a different reason, and a request whose verdict other clients wait on has to stay tracked until that verdict arrives.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

@javenciu
javenciu requested review from a team and KiroCrewCaptain as code owners October 9, 2026 20:53
@javenciu
javenciu requested a review from bolichen97 October 9, 2026 20:53
@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: checking Automated validation is still running labels Oct 9, 2026
@github-actions

github-actions Bot commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5, fork) — ✅ PASS

Premise-level review of 1ddbf2cf1cba87b951d367617520f36558432b67 via the fork AI-review pipeline — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

The fix is at mechanism level and is declared. The Goal line and the title describe an approach the diff dropped.

Notes

  • Item 1, framing: the Goal says "For a tool call that means the entry is retired", and for a lease request "the request is handed off". The diff does neither. It says "This PR does not retire it" and "Rejected: handing a cancelled lease request to a parked rider". The title, "stop tracking a request its client cancelled", is wrong for both halves. Subtraction: none in code; the title is the PR's own text and can be shortened to what ships.
  • Item 3, undeclared in the Goal: the hold is applied "whether or not anything was cancelled" (docs/architecture/mcp.md hunk). So a plain final unsubscribe on an identity server now waits on an in-flight subscribe where it was forwarded before. Its stated cause is the cancel change, but the same race exists on base with no cancel at all.

What this change ships

Inventory (6 items) — 4 justified

Intent: stop a cancelled request from aging a healthy shared MCP backend into the wedge ceiling and recycling every co-pooled session. FIX.

  1. A cancelled resources/subscribe or resources/unsubscribe is no longer forwarded to the server, so the server's answer settles the lease. — justified
  2. A cancelled tool call is still marked and forwarded, as on base (backend_stubs.py:429). — justified
  3. On an identity server, a final unsubscribe is answered locally and its release waits on a same-caller subscribe still in flight. — rides along, applies without any cancel
  4. While that release waits, the URI is orphan-marked, so an update the server sends for it does not count as a hazard. — rides along
  5. The stub bridge drops a cancelled id from _outstanding_ids, so it stops driving liveness pings and gets no -32603 when the bridge drops. — justified
  6. _PendingRequest.release_owed, with settlement in three arms (_on_resource_subscription_response, and both arms of _settle). — justified

[FIRST-PRINCIPLES-REVIEWED] 1ddbf2c

Description read

[DESCRIPTION-READ] 371cd54ecdd07924a776e84787d4f90ba61850ec76575dd9ce5f12c366e8bca2

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5, fork) — ⏭️ skipped

Revision 1ddbf2cf1cba87b951d367617520f36558432b67 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5, fork) — ✅ PASS

Design-level review of 1ddbf2cf1cba87b951d367617520f36558432b67 via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks. Each cancel case now ends in a settled entry or a verdict, and the identity release hold is a one-rule, reversible change.

Notes

  • The identity release hold also changes final unsubscribes that nobody cancelled ("whether or not anything was cancelled"). The description's justification for that sits past the truncation cut, so the take-away readers were judged only on the visible Compatible:/Accepted: lines.
  • The title says "stop tracking", but a cancelled tool call still stays pending and marked (from fix(mcp): park waiting tools off the worker so a wait stalls nobody #18841). Only the stub's _outstanding_ids actually stops tracking it, so the title overstates the backend half.

[DESIGN-REVIEWED] 1ddbf2c

Description read

[DESCRIPTION-READ] 26c860394417a919bc35cfc1c3562daa2d3ba17bdfe67bfc7fef65a069b28d41

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review (fork) — ✅ no blocking findings

Reviewed 1ddbf2cf1cba87b951d367617520f36558432b67 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 1ddbf2c

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 9, 2026
@github-actions

github-actions Bot commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review (fork) — ✅ no blocking findings

Reviewed 1ddbf2cf1cba87b951d367617520f36558432b67 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 1ddbf2c

@javenciu

Copy link
Copy Markdown
Contributor Author

What changed in the second commit

A cancel retired every request it named, so for the resources/subscribe that took a URI's lease it dropped the server's verdict before _settle (backend.py, _route_backend_message). The URI stayed in _lease_awaiting_grant, and every stub parked behind it waited for the backend's whole lifetime. A frame tied to a cancelled tools/call also lost its owner, so on a shared backend a server request recycled it. Now a cancelled subscribe or unsubscribe is handed off exactly as a detaching stub's is: _hand_off_lease_request (new, also called by detach_stub, whose inline logic it replaces) gives it to the first parked rider or to the release sentinel, and the cancel is not forwarded. The server's answer therefore settles the lease, and the stub that cancelled gets no reply. A retired call is kept with its owner in _cancelled_fids, still bounded by _CANCELLED_FIDS_KEPT. _notification_owner and the server-request routing (relatedRequestId) read it, so a progress notification or a sampling request for that call reaches its stub instead of another stub or the recycle. I chose this over keeping cancelled calls in _pending_requests because the heartbeat, the response path and detach all read that table, and each would need an exclusion. The hand-off reuses the detach rule, which already covers a lease request whose stub stopped waiting.

New tests in test/test_mcp_gateway_backend_coverage.py (TestForwardFromStub), red on the first commit (pytest -n 0 -k "cancel or rider_parked_during_grant or frames_tied"): 4 failed, 17 passed.

FAILED ...::test_a_cancelled_first_subscribe_hands_its_verdict_to_the_parked_rider - AssertionError: assert [{'method': '...'gw-4242-1'}}] == []
FAILED ...::test_a_cancelled_subscribe_the_server_never_answers_still_ages_out - AssertionError: assert [] == ['s2']
FAILED ...::test_a_late_grant_for_a_cancelled_subscribe_is_released_not_forwarded - AssertionError: assert {'file:///watched.txt'} == set()
FAILED ...::test_frames_tied_to_a_cancelled_call_reach_its_owner - asyncio.queues.QueueEmpty

The first test fails at its first check, because the cancel was forwarded upstream. The fourth fails because the progress notification never reached its owner.

Controls, passing on both commits: test_cancelled_request_leaves_the_pending_table and test_cancelled_call_does_not_age_into_the_wedge_ceiling (a cancelled tools/call is still retired and never reaches the wedge ceiling), test_rider_parked_during_grant_gets_the_server_verdict (an uncancelled subscribe is unchanged), test_detach_of_inflight_forwarder_promotes_the_parked_rider and test_orphaned_grant_is_released_not_leaked (the detach behaviour the shared helper keeps).

Green: the same selection, 21 passed. The 53 test/test_mcp_gateway*.py files plus test/test_stub_bridge_liveness.py, serially: 1959 passed, 38 skipped. mypy, flake8, isort, scripts/check_black_formatting.py (both changed files are in the black baseline and were not reformatted), COMMENT_HISTORY_BASE_REF=origin/main scripts/check_comment_history.py and ./scripts/docs-lint.sh: all exit 0.

Every path that changes the lease state, checked

Each path that changes _lease_awaiting_grant or hands off a lease request has a test through the real backend:

  • Backend gone: test_backend_gone_after_a_cancelled_first_subscribe_answers_every_rider. _broadcast_backend_gone answers the rider holding the handed-off subscribe and the rider still parked behind it, each under its own id, then clears the lease state. The stub that cancelled gets nothing.
  • Wedge recycle: test_the_wedge_recycle_answers_the_rider_of_a_cancelled_subscribe. A cancelled subscribe the server never answers trips the hard ceiling, and the heartbeat's own recycle answers the rider that holds it.
  • Replay after a restart: test_a_rider_parked_behind_a_replayed_subscribe_still_settles. A replayed subscribe is owned by the release sentinel, so no client cancel can match it. The stub parked behind it gets the server's verdict.
  • Cancelled unsubscribe, and the drain of parked replacements: test_a_cancelled_final_unsubscribe_releases_and_drains_the_replacements. The cancel is not forwarded, and the release still settles. The stub parked behind the release then takes the lease afresh and is answered.

That last test found a gap in the hand-off: on a confirmed release, the sentinel arm took only the release waiters out of routing, never the stub the release was sent for. A detach prunes that stub separately, but a cancel does not, so the stub stayed routed for a lease it had released. A later subscriber would then have been answered locally with no upstream lease behind it. The sentinel arm now also removes pending.origin_stub on a confirmed release. For a detach that removal does nothing, because detach has already pruned the stub. Red before that line, with the other three new tests passing:

FAILED ...::test_a_cancelled_final_unsubscribe_releases_and_drains_the_replacements - AssertionError: assert 's1' not in {'s1'}
1 failed, 3 passed

Green: test/test_mcp_gateway_backend_coverage.py, 345 passed. The 53 test/test_mcp_gateway*.py files plus test/test_stub_bridge_liveness.py, serially, on the final tree: 1963 passed, 38 skipped; every gate above exits 0 again.

Review findings answered

  • GPT 6.1 Review (blocking, backend.py:1634): a cancelled lease-taking subscribe is no longer retired. It is handed to the first parked rider or to the release sentinel and its cancel is not forwarded, so a late reply runs the lease transition for cleanup only, and a server that never answers still ages into the wedge ceiling, whose recycle answers the riders.
  • Opus 5.5 advisory 2: a cancelled call's owner is kept in _cancelled_fids and read by _notification_owner and the relatedRequestId routing, so a late progress notification or server request reaches the stub that made the call, and a shared backend is not recycled for it.
  • First Principles CONCERNS: the cancelled-request memory is kept, because it now carries each cancelled call's owner for that routing. test_frames_tied_to_a_cancelled_call_reach_its_owner is red without it, so it no longer only changes a log level.

@javenciu
javenciu force-pushed the fix/mcp-gateway-drops-cancelled-pending branch from 74731d4 to deab3b3 Compare October 10, 2026 05:33
@javenciu

Copy link
Copy Markdown
Contributor Author

GPT 6.1 Review, blocking finding at backend.py:1629: fixed

The finding holds. After a stub cancels its own resources/unsubscribe, _hand_off_lease_request re-owns the request under the release sentinel and keeps the stub in origin_stub. The in-flight guard in _handle_resource_subscription matched only p.stub_uuid == stub_uuid, so a same-stub resubscribe was no longer refused while the release was in flight. In identity mode that subscribe was forwarded, and an out-of-order grant could then be erased by the sentinel arm's subscribers.discard(pending.origin_stub) on the release. The sentinel arm also left the stub's _grant_callers entry in place on a confirmed release.

Change, in src/kiro_crew/mcp_gateway/backend.py:

  • The in-flight guard in _handle_resource_subscription also matches a sentinel-owned unsubscribe whose origin_stub is the subscribing stub, so the resubscribe is refused locally with the same "still in flight" error an uncancelled unsubscribe gets. A replayed unsubscribe has no origin_stub and is unaffected.
  • The sentinel arm of the unsubscribe response pops _grant_callers[(uri, origin_stub)] on a confirmed release, matching the stub-owned unsubscribe arm. For a detach that pop does nothing, because detach removed the entry already.

Test, in test/test_mcp_gateway_backend_coverage.py: TestForwardFromStub::test_a_stub_cannot_resubscribe_over_its_own_cancelled_unsubscribe. Identity backend; s1 is granted with a caller, unsubscribes, cancels the unsubscribe, resubscribes: refused locally, nothing forwarded. The release settles: _grant_callers is empty and s1 is not routed. A later subscribe is forwarded, granted and routed with its caller again.

Red on the previous head (74731d4b36), the test file copied onto it, 3 serial runs, each:

FAILED test/test_mcp_gateway_backend_coverage.py::TestForwardFromStub::test_a_stub_cannot_resubscribe_over_its_own_cancelled_unsubscribe
E       AssertionError: the resubscribe was forwarded upstream past the in-flight guard
E       assert 3 == 2

Green on this head, 3 serial runs: 1 passed each. The file: 346 passed. The 53 test/test_mcp_gateway*.py files plus test/test_stub_bridge_liveness.py, serially, the changed file first: 1964 passed, 38 skipped (the 53 files plus test/test_stub_bridge_liveness.py). mypy, flake8, isort, scripts/check_black_formatting.py (both files are in the black baseline and were not reformatted), COMMENT_HISTORY_BASE_REF=origin/main scripts/check_comment_history.py and ./scripts/docs-lint.sh: all exit 0.

@javenciu
javenciu force-pushed the fix/mcp-gateway-drops-cancelled-pending branch from deab3b3 to ba453b9 Compare October 10, 2026 06:02
@javenciu

Copy link
Copy Markdown
Contributor Author

Rebased onto current main, with no change to the diff (the patch-id is the same). This clears Semgrep's report on main's own test_readiness_poll_gate_max_age_14857.py, which main added after this PR's base.

@javenciu
javenciu force-pushed the fix/mcp-gateway-drops-cancelled-pending branch 2 times, most recently from 5143b14 to 264cacb Compare October 11, 2026 21:03
@javenciu

Copy link
Copy Markdown
Contributor Author

span=65c967e155eb F1 (backend.py:1788 on 5143b14, a retried identity subscribe losing its lease): fixed in the second commit. On an identity-capable server a subscribe the gateway took over, whether cancelled, retracted, evicted or a detached stub's, is released as its caller when nobody takes the grant, and the server removes whatever that caller then holds. _handle_resource_subscription now refuses a subscribe while _identity_lease_transition_unsettled (src/kiro_crew/mcp_gateway/backend.py:1678) finds such a subscribe, or an unsubscribe, of the same principal for that URI still pending (src/kiro_crew/mcp_gateway/backend.py:3320). The principal is the caller when the gateway names one, so another stub of the same session is held too, and otherwise the stub. The release the sentinel's grant arm sends keeps the subscribe's origin_stub (src/kiro_crew/mcp_gateway/backend.py:2739), and _release_upstream_subscriptions records the caller on every release it sends (src/kiro_crew/mcp_gateway/backend.py:4009), so the refusal lasts until that release settles. The refusal cannot reach a subscribe already on the wire: when two stubs of one session subscribe and one then cancels, detaches or retracts its subscribe with an unsubscribe, the other's subscribe was sent first. So every identity release the gateway sends on its own is decided at one place, _defer_identity_release (src/kiro_crew/mcp_gateway/backend.py:3950, run at src/kiro_crew/mcp_gateway/backend.py:4002): no release is sent for a URI and caller while a stub of that caller is routed for the URI, and while a stub's subscribe of that caller for the URI is pending the release is left to its verdict (release_owed). A grant of that subscribe needs no release; any other verdict sends it (src/kiro_crew/mcp_gateway/backend.py:3586, src/kiro_crew/mcp_gateway/backend.py:2754). The arms for a subscribe the gateway took over hand a named caller's release to the same rule on a grant or an unusable answer (src/kiro_crew/mcp_gateway/backend.py:2694, used at src/kiro_crew/mcp_gateway/backend.py:2730 and src/kiro_crew/mcp_gateway/backend.py:2756), so another session's routed stub never holds it back. That covers the release from the sentinel's verdict arms (after a cancel, detach, retract or evict), from detach and evict themselves, and from an unsettled identity verdict. test_a_resubscribe_cannot_overtake_the_release_of_an_unwanted_identity_grant (test/test_mcp_gateway_backend_coverage.py:1060) plays an identity server that answers in write order and releases by caller. Its same-stub, same-session and after-detach cases fail on 5143b14 in 3 of 3 runs (s1 is routed for the URI, but the server holds no subscription for its caller) and pass here. The other-session case, which must not be held, passes on both. On the same server, test_a_release_waits_for_a_same_session_subscribe_already_in_flight (test/test_mcp_gateway_backend_coverage.py:1130) plays the sibling order with a cancel, a detach and a retract: each fails on 5143b14 in 3 of 3 runs (s2 is routed for the URI, but the server holds no subscription for its caller) and passes here, and its refused-sibling case checks that the release is still sent once. test_a_release_left_to_a_subscribe_that_is_then_given_up_is_sent_once (test/test_mcp_gateway_backend_coverage.py:1175) cancels the second subscribe too: granted or refused, exactly one release is sent, and the granted case fails on 5143b14 with 2 releases for one subscription. test_another_sessions_routed_stub_does_not_hold_back_an_owed_release (test/test_mcp_gateway_backend_coverage.py:1211) adds a stub of another session that is routed when that second grant arrives, once with every answer in write order and once with that stub answered first: the release is still sent once and the server keeps only that stub's subscription. test_a_taken_over_grant_is_released_beside_another_sessions_stub (test/test_mcp_gateway_backend_coverage.py:1260) has a stub of another session routed while a stub cancels or detaches its subscribe, which the server grants or answers with neither result nor error: one release is sent and the server keeps only the other session's subscription. With either arm back to its previous condition, only its two cases fail, 3 of 3 runs (the server still holds a subscription nobody is routed for); on main both detach cases fail the same way. A stub's own resources/unsubscribe is the client's request and is still forwarded as on main (src/kiro_crew/mcp_gateway/backend.py:3424): when another stub of the same session already has a subscribe for the URI on the wire, that unsubscribe removes the subscription the other stub's grant then routes. main has the same gap and this PR does not change that path; the description says so.

span=65c967e155eb F2 (backend.py:1647 on 5143b14, tombstones keeping whole requests): fixed in the second commit. _cancelled_fids now holds a _CancelledRequest (src/kiro_crew/mcp_gateway/backend.py:382) with only what its readers use: the owning stub, read by _notification_owner and the relatedRequestId routing of server requests, and the progress token, read by _notification_owner. The response arm reads only the forward id. _retire_cancelled builds it (src/kiro_crew/mcp_gateway/backend.py:1726), and _bounded_progress_token (src/kiro_crew/mcp_gateway/backend.py:365) keeps a token only as a 64-bit number or a string of at most 256 characters, so each entry is bounded and the table stays capped at _CANCELLED_FIDS_KEPT. Arguments, the retry frame and every other field go with the cancel. test_a_cancelled_calls_tombstone_keeps_only_its_owner_and_a_bounded_token (test/test_mcp_gateway_backend_coverage.py:929) cancels two calls carrying 1 MiB of arguments, one with a 1 MiB token. It fails on 5143b14 in 3 of 3 runs (a cancelled call's tombstone kept its request payload) and passes here.

span=6216a66100d7 F3 (test/test_stub_bridge_liveness.py:661, EOF after polls run out): fixed in the test. The fake gateway writer sets an event when the forwarded cancel arrives (test/test_stub_bridge_liveness.py:649). The closer waits on it for at most 5 s and, if it never comes, raises with the frames the gateway did see (test/test_stub_bridge_liveness.py:668). feed_eof() runs only after the event (test/test_stub_bridge_liveness.py:670). There is no polling and no sleep. With the stub changed to drop the cancel without forwarding it, the new test fails in 3 of 3 runs (the cancel never reached the gateway), where the previous version passes in 3 of 3 runs.

span=910aa765b5df (Opus, the last-stub recycle): not changed in this round. The finding is accurate. A call its client cancelled leaves _pending_requests, so the teardown check has_recyclable_in_flight(backend._pending_requests, stub_uuid) (src/kiro_crew/mcp_gateway/daemon/connection.py:658) no longer counts it. A server that ignores the cancel then keeps running that work after the last stub leaves, where main recycled the backend. The fix you describe is to count the stub's unanswered, non-exempt tombstones in that check, which needs a backend helper and a change to daemon/connection.py, a file this PR does not touch. It is left for the maintainers to decide whether it belongs in this PR.

span=09845c8a83b5 (First Principles, item 6): same gap as the Opus finding above, and not changed in this round. The description's claim that the backend "already received the cancel" is true, but it does not show that a server honours it, so it does not justify skipping the recycle. The clear condition that fits is the second one: a cancelled entry that the server has not answered still counts for the Scope B recycle. The tombstone now has a bounded place for that, and the change needs only a "recycle-exempt" flag plus the helper above.

@javenciu
javenciu force-pushed the fix/mcp-gateway-drops-cancelled-pending branch from 264cacb to 6ed9940 Compare October 11, 2026 22:31
@javenciu

Copy link
Copy Markdown
Contributor Author

span=65c967e155eb F1 (backend.py:3974 on 264cacb, a release not left to a same-caller replay): fixed in the second commit, for every kind of in-flight subscribe rather than replays alone. _defer_identity_release (src/kiro_crew/mcp_gateway/backend.py:3950) now leaves a release for a URI and caller to any subscribe of that caller for the URI still in flight, whoever owns it (src/kiro_crew/mcp_gateway/backend.py:3975): a stub's own subscribe, as before; a respawn replay for a stub (replay_stub, written at src/kiro_crew/mcp_gateway/backend.py:3783); and a subscribe the gateway took over after a cancel, detach, retract or eviction, or a replay an eviction scoped to release on its grant. The verdict arms already settle the debt for each kind. A grant that routes a stub needs no release (a replay's grant routes its stub while it is attached, src/kiro_crew/mcp_gateway/backend.py:2706); a grant that routes nobody releases through the same rule (src/kiro_crew/mcp_gateway/backend.py:2730); a refusal sends the owed release (src/kiro_crew/mcp_gateway/backend.py:2754, src/kiro_crew/mcp_gateway/backend.py:3586). The drain's subscribe and the coalesced riders exist only on servers without caller identity, and the drain's pending names no caller, so the rule never meets them. test_a_release_waits_for_a_same_caller_replay_in_flight (test/test_mcp_gateway_backend_coverage.py:1302) plays your ordering: A and B replay as one caller and C as another, A detaches, and the server answers C, A, B. It fails on 264cacb in 3 of 3 runs (B is routed for the URI, but the server holds no subscription for its caller: a release landed after its grant) and passes here; it also passes on main, whose release check counted C as routed. test_a_release_waits_for_a_same_caller_subscribe_that_routes_nobody (test/test_mcp_gateway_backend_coverage.py:1334) covers the two kinds whose grant routes nobody, a cancelled subscribe and a replay scoped by an eviction: on 264cacb the release went out at once and that grant sent a second (2 releases for one subscription), and here one is sent. With one kind dropped from the rule, only that kind's cases fail, 3 of 3 runs: the stub's own (the six cases from the earlier rounds), the replay (your ordering), or the two that route nobody.

@javenciu
javenciu force-pushed the fix/mcp-gateway-drops-cancelled-pending branch from 6ed9940 to e22f99b Compare October 12, 2026 00:02
@javenciu

Copy link
Copy Markdown
Contributor Author

span=65c967e155eb (backend.py:1727 on 6ed9940, owners kept whole in cancellation tombstones): fixed in the second commit, for every field a tombstone keeps. There was no limit on a stub id to reuse: src/kiro_crew/mcp_gateway/daemon/connection.py:175 reads it from the Register frame, and the only checks are that it is non-empty (src/kiro_crew/mcp_gateway/daemon/connection.py:254) and has no reserved prefix (src/kiro_crew/mcp_gateway/daemon/connection.py:270). The new _CANCELLED_OWNER_MAX_LEN (src/kiro_crew/mcp_gateway/backend.py:367, 256 characters; a real stub registers a 36-character uuid4) sits beside the progress token's limit (src/kiro_crew/mcp_gateway/backend.py:362). _retire_cancelled still removes the pending request and forwards the cancel, then keeps no tombstone when the owner is longer (src/kiro_crew/mcp_gateway/backend.py:1732); the owner is never truncated. A tombstone now holds a gateway-minted forward id as its key, an owner of at most 256 characters and a progress token bounded by _bounded_progress_token, and at most 256 tombstones are kept. When an owner is past the bound, its call's late frames are handled as for an unknown request. The response is dropped with the response to unknown id warning, and a progress notification is dropped as unattributable, as one echoing an over-long token already is. A server request takes the existing fallback for one the gateway cannot attribute: it goes to the only attached stub, and with any other number of stubs the backend is recycled. test_a_cancel_keeps_no_tombstone_for_an_owner_past_its_bound (test/test_mcp_gateway_backend_coverage.py:986) plays your case with the cap at 2: s1 cancels a call, then three stubs with distinct owners of 257 characters or 1 MiB do the same. Only s1's tombstone is kept, every cancel reaches the server, and a late progress notification for one of the three reaches no stub. It fails on 6ed9940 in 3 of 3 runs (a cancel kept a tombstone for an owner past the bound) and passes here. test_an_owner_within_its_bound_keeps_its_tombstone_and_its_late_frames (test/test_mcp_gateway_backend_coverage.py:1028) checks that an owner of 36 characters, and one of exactly 256, keeps its tombstone, and that its late progress notification and server request still reach it; it passes on both heads. With the bound dropped, only the two cases past it fail; with it off by one, only the case at 256 fails; 3 of 3 runs each.

@javenciu
javenciu force-pushed the fix/mcp-gateway-drops-cancelled-pending branch from e22f99b to e93d496 Compare October 12, 2026 02:30
@javenciu

Copy link
Copy Markdown
Contributor Author

span=65c967e155eb (backend.py:4097 on e22f99b, a cancelled call's routing outliving its stub): fixed in the second commit, at the one place a stub stops being attached. detach_stub is the only code that removes a stub's inbox (src/kiro_crew/mcp_gateway/backend.py:1508), and attach_stub refuses a uuid while its inbox is there (src/kiro_crew/mcp_gateway/backend.py:1323), so a uuid can be accepted again only after that removal. Every departure goes through it: a disconnect, including after a backend death or recycle (src/kiro_crew/mcp_gateway/daemon/connection.py:668), a full inbox, the tool-surface probe, the app-call stub and the respawn swap. detach_stub now drops the departing stub's tombstones in the same locked step that removes its inbox (src/kiro_crew/mcp_gateway/backend.py:1514), so no attach can find the uuid free while a tombstone still names it. After that, the call's late frames are handled as main handles a departed stub's: the response logs the unknown-id warning, a progress notification is dropped as unattributable, and a server request goes to the only attached stub or recycles a shared backend. A warm-pool rekey (evict_stub_subscriptions) keeps the stub attached and, as on main, leaves its calls routed to it, so a tombstone follows its call there as the pending entry did; that path is unchanged. test_a_departed_stubs_cancelled_call_does_not_reach_a_stub_reusing_its_uuid (test/test_mcp_gateway_backend_coverage.py:760) plays your ordering: s1 cancels a call and detaches while s2 keeps the backend, a new connection attaches as s1, and the call's late progress notification and server request reach neither it nor s2. It fails on e22f99b in 3 of 3 runs (a departed stub's cancelled call reached a stub reusing its uuid) and passes here and on main. test_another_stubs_departure_keeps_a_cancelled_calls_routing (test/test_mcp_gateway_backend_coverage.py:795) checks that another stub's departure leaves an attached stub's tombstone and late frames alone. With the cleanup dropped, only the first test fails; with every tombstone dropped on any detach, only the second does; 3 of 3 runs each.

@github-actions github-actions Bot added the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Oct 12, 2026
@javenciu
javenciu force-pushed the fix/mcp-gateway-drops-cancelled-pending branch from e93d496 to b69aaa7 Compare October 12, 2026 08:51
@javenciu

Copy link
Copy Markdown
Contributor Author

Rebased onto main 0a47021. It conflicted with #18841 in the cancel branch of Backend.forward_from_stub (src/kiro_crew/mcp_gateway/backend.py:1925-1942): #18841 marks a cancelled entry (_PendingRequest.cancelled) and has _heartbeat_once skip it, where this change retires the entry. I kept retirement and removed the mark, its skip and its paragraph in docs/architecture/mcp.md; #18841's tests of the mark now assert retirement (test/test_mcp_gateway_inflight_report.py::TestCancelledCall, test/test_gatewayd_more_coverage.py::TestDisconnectTeardown::test_forwarded_cancel_then_disconnect_recycles_only_for_other_work).

Behaviour that differs from main as a result: a stub that cancels a call and then disconnects no longer recycles the backend on that call's account (another call of the stub still in flight does). If you want the recycle kept after a client cancel, the tombstone in _cancelled_fids can drive it; say so and I will add it.

A correction to the description: the wedge-ceiling symptom in Problem / Motivation no longer reproduces on main, because #18841's mark already skips a cancelled entry (test_cancelled_call_does_not_age_into_the_wedge_ceiling passes on main now). What this change still fixes is the entry kept per cancel until the stub detaches, a cancelled resources/subscribe or resources/unsubscribe stranding a URI's lease, the late frames of a cancelled call, and the stub's liveness monitor. The code and tests of the second commit are unchanged.

@github-actions github-actions Bot removed the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Oct 12, 2026
@github-actions github-actions Bot added the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Oct 17, 2026
@bolichen97

Copy link
Copy Markdown
Contributor

This branch conflicts with main. Please rebase it, and the reviewer will look again at the identity-release part.

@javenciu
javenciu force-pushed the fix/mcp-gateway-drops-cancelled-pending branch from 016bbc0 to f43e898 Compare October 18, 2026 05:54
@javenciu

Copy link
Copy Markdown
Contributor Author

@bolichen97 Rebased. The branch is now one commit on main at c3b65e8, head f43e898850ef7e0ae493bf70841f56037e3ba322.

  1. The conflict came from the stub.py split (refactor: split mcp_gateway stub.py and rewriter.py under the line cap #21534), which moved run_bridge into stub_bridge.py. The notifications/cancelled branch moved there unchanged, right after _outstanding_ids.add (src/kiro_crew/mcp_gateway/stub_bridge.py:457-463). stub.py now matches main exactly, and the moved hunk's added lines match the old stub.py hunk line for line.

  2. The identity-release patch is unchanged. Comparing the old diff's added and removed lines with the new diff's for the five backend files:

    F="src/kiro_crew/mcp_gateway/backend.py src/kiro_crew/mcp_gateway/backend_leases.py src/kiro_crew/mcp_gateway/backend_routing.py src/kiro_crew/mcp_gateway/backend_stubs.py src/kiro_crew/mcp_gateway/backend_wire.py"
    diff <(git diff 12835a82ce 016bbc02d0 -- $F | grep '^[+-]') <(git diff c3b65e8f06 f43e898850 -- $F | grep '^[+-]'); echo RC=$?
    RC=0
    

    The diff prints nothing, and each side is 94 insertions and 17 deletions. So the reviewer reads the same _hold_identity_release and release_owed code that was reviewed at 016bbc0.

  3. One layout change beyond the move. Main's copy of test/test_mcp_gateway_backend_coverage.py is black-clean, and test(mcp-gateway-connections): replace fixed waits with signal waits #21209 took it out of the black baseline, so the black gate now checks the whole file. This PR's added tests there are re-wrapped with the pinned black 26.3.1. Only spaces, line breaks and trailing commas change, so the tests send and assert the same things. No line that main has in that file changes (0 deletions against main). This is why that file's diff grows from +356 to +481 lines.

  4. Tests on the new head: pytest test/test_mcp_gateway_backend_coverage.py test/test_stub_bridge_liveness.py test/test_backend_bindings.py -p no:cacheprovider -q -n 8 gives 527 passed in each of 3 runs, RC=0. I re-ran the red check on c3b65e8. With main's copies of the six source files this PR changes copied over this branch's, the description's -k selection gives 11 failed, 5 passed, and the 11 failures are the cases the description lists. With main's stub_bridge.py copied in, test_a_cancelled_request_is_no_longer_outstanding fails on assert ['call-7', 'call-8'] == ['call-8']. flake8, isort, the black gate and the comment-history gate pass.

  5. The description's bridge line now names stub_bridge.py. The mcp-gateway: route detach, evict and the sentinel grant arm through _hold_identity_release #21433 follow-up link is unchanged.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 18, 2026
@javenciu

Copy link
Copy Markdown
Contributor Author

span=1366e2d6f925

  • The Goal promises things the diff doesn't do. kept: the review is right, and this rebase changes neither side. Tool-call entries stay pending and marked (src/kiro_crew/mcp_gateway/backend_stubs.py:422-431). A cancelled subscribe or unsubscribe is not handed off: its cancel is not forwarded, and the server's verdict settles it under the stub that sent it (backend_stubs.py:409-421). The description's "What changed" section states what ships. The Goal is frozen, so rewriting it, or accepting the mismatch, is the maintainer's call (#issuecomment-6218980971).

@javenciu

Copy link
Copy Markdown
Contributor Author

span=d6a3249cd1be

  • The release hold changes more than the cancel path. kept: it does, and the description says so in "Backwards compatibility". On an identity-capable server, a stub's final unsubscribe sent while a same-caller subscribe for that URI is in flight is answered locally, and its release waits for that subscribe's verdict, cancelled or not (src/kiro_crew/mcp_gateway/backend_leases.py:436-452). It can't ship on its own PR, because the cancel change makes it necessary. That cancel is no longer forwarded (backend_stubs.py:409-421), so a server that used to drop the cancelled unsubscribe now answers it. Forwarding the release then would let the server grant the in-flight subscribe and drop the caller's one subscription. A grant discharges the held release, and a refusal or an unsettled verdict sends it (backend_leases.py:646-653, backend_routing.py:516-524, :547-548). The reviewer bolichen97 named for the identity-release part reads it at this head. This rebase leaves those lines unchanged.

@javenciu

Copy link
Copy Markdown
Contributor Author

span=d1db48547342

  • The take-away check is capped at CONCERNS. kept: the "Backwards compatibility" section runs to byte 8225 of the description, past the 8000 the review reads, and a rebase doesn't change that. The two readers the cut can hide are listed in full at the previous head (#issuecomment-6218982323). The four reader lines there are still accurate: the five backend files are byte-identical to that head. Making the section fit means cutting text the frozen Goal and "What changed" depend on, so it stays as it is.

@javenciu

Copy link
Copy Markdown
Contributor Author

span=9ad5f4afe93e

  • Item 3, the identity-release hold rides along. kept: this commit is a rebase. The five backend files have the same added and removed lines as at 016bbc0, so the answer posted there for this item still holds (#issuecomment-6218984749). The hold is decided when the unsubscribe arrives, before any cancel, as the item says (src/kiro_crew/mcp_gateway/backend_leases.py:442-444). It ships with the cancel change because that cancel is no longer forwarded (backend_stubs.py:409-421). A server that used to drop the cancelled unsubscribe now answers it, and forwarding the release beside a same-caller subscribe still in flight would leave the new subscriber routed to nothing (backend_leases.py:436-440). test_a_last_holder_unsubscribe_waits_for_a_same_caller_subscribe[granted|refused] (test/test_mcp_gateway_backend_coverage.py:815-898) drives that case with the cancel in it. The item clears when the Goal is amended, which only a maintainer can do.

@javenciu

Copy link
Copy Markdown
Contributor Author

span=ee2ec21693e6

  • Item 4, the Goal says a cancelled tool call's entry is retired. kept: the item is right, and this rebase doesn't change it. A cancelled tool call is still remapped, forwarded and marked, as on main (src/kiro_crew/mcp_gateway/backend_stubs.py:422-431), and the description's "What changed" section says this PR does not retire it. The Goal is frozen, so the item clears only when a maintainer amends it, as the earlier answer said (#issuecomment-6218980971).

@javenciu

Copy link
Copy Markdown
Contributor Author

span=b08c7e8b860d

  • Detach and evict still ignore a same-caller subscribe in flight. kept: the finding is accurate, and that code is main's. detach_stub (src/kiro_crew/mcp_gateway/backend_stubs.py:216-221) and evict_stub_subscriptions (backend_leases.py:221-226) still count only routed stubs of the caller, and this diff has no hunk in either. No cancel reaches them. The one site a cancel does reach, a stub's final unsubscribe, is the one this PR guards (backend_leases.py:442-444). Sending detach, evict and the sentinel's un-owed arm through _hold_identity_release (backend_leases.py:583) is tracked as mcp-gateway: route detach, evict and the sentinel grant arm through _hold_identity_release #21433, which the description links. It changes release timing on paths that have no cancel, so it stays out of this change.

@github-actions github-actions Bot removed the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Oct 18, 2026

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks correct to me, with two should-fix items. Head reviewed: f43e898.

Thanks for the rebase. It now applies cleanly on main, and I did a full review of this head.

What I checked:

  • The cancel branch in forward_from_stub returns before the remap, so no frame for a cancelled subscribe or unsubscribe goes upstream. Nothing skips the caller identity strip, because nothing is written.
  • _settle pops the pending entry before _on_resource_subscription_response and the sentinel arm run. So _hold_identity_release never finds the verdict it is settling, and a refused subscribe with release_owed cannot hand the obligation to itself.
  • The grant arm, the refused arm and the unsettled arm all settle an owed release through one rule. The sentinel orphan arm does the same on a grant and on a refusal. A release with no caller is never held, so servers without caller identity behave as on main.
  • run_bridge discards a cancelled id only on notifications/cancelled with a dict params, so a malformed cancel changes nothing.
  • No mcp_gateway file changed on main since the merge base, so main does not already carry this fix.
  • The follow-up for detach, evict and the sentinel grant arm is tracked in #21433.

Not repeated here: the Goal wording versus the shipped design (Design Review), and the scope of the identity release hold (First Principles Review). You already answered both on this head. The hold also changes a final unsubscribe that was never cancelled. That scope call belongs to a maintainer, not to this review.

Backend test shards were still running when I read the checks, so I did not count them.

Two should-fix items are inline: the new tests grow a test file that is already over its line cap row, and the architecture doc does not describe the new release hold.

assert _frames(backend) == [["not", "a", "dict"]]

@pytest.mark.asyncio
async def test_a_hung_backend_whose_only_call_was_cancelled_is_recycled(self) -> None:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 The new cases grow a file that is already past its line cap row

This file has 6,525 lines on main. Its row in .github/file-line-cap-baseline.txt is 5,650. This PR adds 481 lines and takes it to 7,006. scripts/check_file_line_cap.py only reports a baselined file that grew, so CI stays green. But the script states the rule: "It must not grow, and it should be split".

    @pytest.mark.asyncio
    async def test_a_hung_backend_whose_only_call_was_cancelled_is_recycled(self) -> None:

Why it matters: Each PR that adds to this file makes the later split larger, and the cap stops meaning anything for it.

Suggestion: Move the nine new test functions and the three helpers (_releases, _fid_of, _answer) into a new file, for example test/test_mcp_gateway_backend_cancel.py. Import _make_backend, _frames, _line, _drain and _settle_lease from the existing module, or from a shared helper module if one exists. The diff to this file then becomes zero lines.
Rule: AGENTS.md (no hand-written file over 2,000 lines); scripts/check_file_line_cap.py "The baseline".

Comment thread docs/architecture/mcp.md
stub leaving after a cancel still recycles a backend whose synchronous tool kept
running. A subscribe or unsubscribe is not marked, because its verdict settles
the shared lease tables. The client side is gated on the declaration because a
the shared lease tables: its cancel is not forwarded at all, so the server

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 The spec describes the cancel change but not the new identity release hold

The new paragraph says why a cancelled subscribe or unsubscribe is not forwarded. It does not describe the second behaviour change in this PR. On an identity capable server, a final unsubscribe sent while a same caller subscribe for that URI is in flight is now answered locally. Its upstream release then waits for that subscribe's verdict (release_owed, _hold_identity_release). This also applies when nothing was cancelled. Today only the PR description and the code comments say so.

the shared lease tables: its cancel is not forwarded at all, so the server
answers it and the verdict settles the lease under the stub that sent it, as on
main when a server answers a request it was asked to cancel.

Why it matters: A reader of the owning doc cannot learn that a final unsubscribe can be answered without a server round trip. That reader also cannot learn who sends the deferred release. #21433 will extend the same rule to detach and evict, so the doc gap will grow.

Suggestion: Add two or three sentences after this paragraph. State the rule as _hold_identity_release applies it: the release is held while another stub of the caller is routed for the URI, or while a same caller subscribe is in flight. That subscribe inherits the release. A grant discharges it, and any other verdict sends it unless the rule still holds. While you edit here, wrap the line that ends in "The client side is gated on the declaration because a" to match the paragraph.
Rule: AGENTS.md "Specs and docs" (owning spec updated in the same commit); AGENTS.md "Read before you touch" maps mcp_gateway/ to this doc.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 18, 2026
@javenciu
javenciu force-pushed the fix/mcp-gateway-drops-cancelled-pending branch from f43e898 to fc3f664 Compare October 19, 2026 01:59
@javenciu

Copy link
Copy Markdown
Contributor Author

Thanks for the review. Both items are done in this push.

  1. test/test_mcp_gateway_backend_coverage.py:636: the nine new test functions and the helpers _releases, _fid_of and _answer moved to a new file, test/test_mcp_gateway_backend_cancel.py (533 lines), in two classes, TestForwardFromStubCancel and TestIdentityReleaseHold. Each body is unchanged: the 455 moved test lines and the 23 helper lines are identical to the old file under diff. This PR's diff to test/test_mcp_gateway_backend_coverage.py is now zero lines (git diff origin/main HEAD -- test/test_mcp_gateway_backend_coverage.py prints nothing). The new file imports _make_backend, _frames, _line, _drain, _settle_lease and the autouse _no_real_metrics_file from test/mcp_gateway_backend_coverage_helpers.py, the way main's test/test_mcp_gateway_backend_coverage.py does since test: split test_mcp_gateway_backend_coverage.py by behaviour under test #21866, so the tests pass on main after that split: 13 passed in each of 3 runs, and 282 passed in each of 3 runs across the new file, test/test_stub_bridge_liveness.py, test/test_backend_bindings.py and main's test/test_mcp_gateway_backend_coverage.py (87 of them). Against main's own src the new file gives 11 failed, 2 passed.

  2. docs/architecture/mcp.md:4810: added a paragraph after the in-flight report paragraph, now at docs/architecture/mcp.md:4881-4890:

    On an identity-capable server, the upstream release of a caller's subscription to a URI is held while another stub of that caller is routed for the URI, or while a same-caller subscribe for it is in flight (Backend._hold_identity_release), whether or not anything was cancelled. The unsubscribing stub is answered locally, and with no such stub routed the first same-caller subscribe in flight inherits the release (_PendingRequest.release_owed). A grant that routes a stub discharges it, because the caller's one upstream subscription now serves that stub, whose own departure releases it. A refusal, an unknown verdict or a grant that routes no stub has the gateway send it as that caller, unless the rule still holds.

    The line that ended "The client side is gated on the declaration because a" is re-wrapped to the paragraph's width, now docs/architecture/mcp.md:4877-4879.

No src/ change since the reviewed head f43e898: the added and removed lines of git diff <old merge base> f43e898850 -- src and of git diff origin/main HEAD -- src are identical (118 lines each, and diff between them is empty).

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: passed Eligible automated validation passed for the current revision labels Oct 19, 2026

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me. Head reviewed: fc3f664.

Both items from my review of f43e898 are fixed in this push:

  • The cancel and identity release tests moved to the new file test/test_mcp_gateway_backend_cancel.py (533 lines). test/test_mcp_gateway_backend_coverage.py is no longer in the diff, so it does not grow past its line cap row. The new file imports the shared fixture and helpers from mcp_gateway_backend_coverage_helpers.py, and the autouse _no_real_metrics_file fixture is imported, so the moved tests keep their isolation.
  • docs/architecture/mcp.md now describes the identity release hold: when the release is held, who inherits it (release_owed), and how a grant, a refusal, an unknown verdict or a grant that routes no stub settles it. The stray long line in the paragraph above is wrapped.

What I checked on this head:

  • The source hunks in the mcp_gateway package are the same as at f43e898. The only other code change is the new _hold_identity_release row in test/test_backend_bindings.py. My earlier correctness read of the cancel branch, the settle order and the three release_owed arms still applies.
  • No commit on main since the merge base touches the mcp_gateway package, docs/architecture/mcp.md or the touched tests, so main does not already carry this fix.
  • One commit, base main. No touched path is on the security path list.

Not repeated here: the Goal wording and the scope of the identity release hold (Design Review and First Principles Review). You answered both on an earlier head, and the scope call belongs to a maintainer.

I am not approving yet. Backend Tests, Lint, E2E and the Opus and GPT review lanes were still running on this head when I read the checks, and PR Readiness is pending.

javenciu and others added 2 commits October 25, 2026 03:21
…ler subscribe

When the identity release hold answers a last holder's unsubscribe locally,
routing for the URI empties while the caller's subscription is still live
upstream. An update the server sends before the in-flight subscribe's
verdict then routed nowhere and was charged to a correct server as an
unattributable-notification hazard. Orphan-mark the URI, as detach does;
a grant or a confirmed release clears the mark.
@iamwhatever

Copy link
Copy Markdown
Contributor

Rebased onto current main (no conflicts) and addressed the Opus blocking finding at 3a0db53 in 1ddbf2c: when the identity release hold empties routing for the URI, backend_leases.py now orphan-marks it with the same bounded add the detach path uses, so a notifications/resources/updated for the caller's still-live subscription is dropped without recording HAZARD_UNATTRIBUTABLE_NOTIFICATION. A grant clears the mark (existing arms), and a confirmed release clears it too. New test test_an_update_while_the_held_release_waits_is_not_a_hazard fails on 3a0db53 and passes here; it also checks that after the refused subscribe sends the held release and the release succeeds, an update is a hazard again. docs/architecture/mcp.md notes the mark. Local run: test/test_mcp_gateway*.py, test/test_stub_bridge*.py, test/test_backend_bindings.py, 2213 passed.

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me. Head reviewed: 1ddbf2c.

The delta since my review of fc3f664 is the rebase plus one new commit, which orphan-marks a URI when the identity release hold answers the last holder's unsubscribe locally.

What I checked in the new commit:

  • backend_leases.py: the mark is added only when routing for the URI empties, and it uses the same bounded add (_ORPHANED_LEASES_MAX) as the detach path.
  • The mark clears on each way the held release settles. A grant of the in-flight subscribe clears it in _on_resource_subscription_response, and a grant on a sentinel pending clears it in _settle. A refusal or unknown verdict sends the release, and a successful release clears it. A refused release keeps it, which matches the existing detach behavior.
  • A refusal with another same-caller subscribe still in flight passes the obligation on through _hold_identity_release, so the mark correctly stays.
  • The new test asserts the mark directly, so by reading it goes red without the change. It also asserts no hazard while the release waits, and it asserts the hazard comes back after the release succeeds.
  • docs/architecture/mcp.md describes the new mark in the identity release hold paragraph.

All five model review lanes pass on this head, and Security Scope Review finds no security path touched.

I am not approving yet. Backend Tests and one E2E job were still running on this head when I read the checks, PR Readiness is pending, and an earlier change request from another reviewer is still open.

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving head 1ddbf2c.

Gates checked on this head:

  • PR Readiness is passed, and every check on this head is green.
  • All model review lanes pass on this head, with no blocking finding.
  • No file is on the security path list. The security checklist answers are all no: no auth change, no new external input parsing, no trust boundary or tool visibility change, no sandbox change, no approval gate change, and no secret handling.
  • Base is main, and the branch merges cleanly.

My review of the code itself is in my earlier comment on this same head. An earlier change request from another reviewer is still open, so the merge waits for that reviewer.

@iamwhatever

Copy link
Copy Markdown
Contributor

HOLD at 1ddbf2c.

The code looks sound. A cancelled resources/subscribe or unsubscribe is no longer sent its cancel, so the server's verdict settles the lease, and other cancels are marked and forwarded as on main (backend_stubs.py#L408-L431). The new orphan mark while a held release waits is added only when routing empties and is cleared on grant or confirmed release (backend_leases.py#L436-L461). The follow-up from our r107 review is tracked in #21433, which names the three sites, so that request is satisfied. All lanes pass and every check is green.

What still needs a maintainer:

  1. The frozen Goal and the title still say a cancelled tool call is retired and a lease request is handed off. The diff does neither, and the author says only a maintainer may amend the Goal. A maintainer should amend it or accept the mismatch.
  2. First Principles item 3: the identity release hold (_hold_identity_release) also changes a final unsubscribe that nobody cancelled. A maintainer should confirm it ships here.
  3. The head commit 1ddbf2c was pushed from this reviewing account, so it needs a review from someone other than its author. bolichen97's approval covers the code; a maintainer sign-off on points 1 and 2 clears the hold.

Auto-merge is armed, so our earlier change request is what holds the merge.

— crew-pr-reviewer (r125)

@bolichen97

Copy link
Copy Markdown
Contributor

@iamwhatever your change requests here look addressed at the current head 1ddbf2c. Your later review asked for an issue tracking the detach, evict and sentinel grant work that was deferred, and #21433 now tracks it and is linked in the description. The refused resubscribe concern from your first review went away with the hand off design, as your later review noted, and I signed off on the identity release code in my approval on this same head. All checks are green and the branch merges cleanly. If you agree, could you review it again or withdraw the change request?

@iamwhatever iamwhatever left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Recommend APPROVE at 1ddbf2cf1cba87b951d367617520f36558432b67. This approval replaces crew-pr-reviewer's earlier change requests (r107 at 016bbc0 and the deep re-check at 0b765e2).

The fix is sound. A cancelled resources/subscribe or unsubscribe is no longer sent its cancel, so the server's verdict settles the lease under the stub that sent it. Every other cancel is still marked and forwarded as on main (backend_stubs.py#L409-L431). On an identity server, _hold_identity_release (backend_leases.py#L592) holds a release only while a stub or in-flight subscribe of the same caller still depends on it. The release is settled on grant, refusal and unknown verdicts (#L655, backend_routing.py#L516, #L547). The bridge drops a cancelled id from _outstanding_ids (stub_bridge.py#L457).

Our r107 request is met. #21433 is open, linked in the description, and names the detach, evict and sentinel-grant sites. The head commit was pushed from the crew-pr-reviewer account, and maintainer bolichen97 reviewed and approved that exact head, including the identity release code. All lanes pass (UX skipped), PR Readiness and every check are green, there's no third-party change request, and the branch merges cleanly with main.

AI findings judged non-blocking

  • First Principles item 1 / Design note 2: Goal line and title describe retire/hand-off approaches the diff dropped. Description wording only; the body's What changed states plainly that a cancelled tool call stays marked and the hand-off was rejected, and maintainer bolichen97 approved this head with that known. code
  • First Principles item 3 / Design note 1: identity release hold also changes a final unsubscribe nobody cancelled. Declared in Backwards compatibility and docs/architecture/mcp.md; the hold only defers a release while a same-caller stub or subscribe still depends on the subscription, every verdict arm settles it, and maintainer bolichen97 signed off on the identity release code. code

— crew-pr-reviewer (r134)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants