Repository navigation
fix(mcp): send the caller block's session token on the tool-policy read - #12445
Conversation
The tools/call dispatch checks the session's tool policy on the read loop, before the worker installs the caller ContextVar. The policy resolver read the per-session token from that ContextVar, so at the moment it mattered the token was always empty and the loopback request went out with X-Session-Key and no X-Session-Token. A pooled control-plane backend has the token nowhere else: gatewayd spawns it from the daemon's environment and hands the token over per frame in the caller block. Since the gateway began requiring an attestation behind a declared session key, every such read was answered 409 member_identity_unavailable, the client read every 409 as policy_unreadable, and every kirocrew-core / kirocrew-cron tool call on a pooled host was refused with a message pointing at the agents directory. _resolve_tool_policy now takes the caller's token explicitly, handed in by _caller_tool_policy from the CallerContext it already receives; the ContextVar stays as the fallback for callers that arrive with it set. The 409 arm reads the body's code: member_identity_unavailable becomes its own reason, identity_unattested, with its own audit event and an error text that names the missing token. It stays in the refusing set, so nothing an operator excludes becomes unenforced; only the diagnosis changes. The unreadable-spec refusal keeps its reason and drops the misleading "Retry shortly". Fixes #12441
|
Intent: A pooled control-plane MCP backend proves which session it acts for when it reads that session's tool policy, so the gateway's attestation gate (#12164) no longer refuses every tool call on a pooled host; and when the gateway does refuse for want of that proof, the refusal says so instead of blaming an unreadable agent spec. |
Design Review (Fable 5) — ✅ PASSDesign-level review of Design-Verdict: PASS Verify the deferred live check: on the diagnosed pooled host, after gateway restart, tool calls recover and the audit shows no [DESIGN-REVIEWED] ed15d5f |
GPT 5.6 Review — ✅ no blocking findingsGPT 5.6 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
First Principles Review (Fable 5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS The fix is pinned only by unit tests — confirm on the live pooled host that a restarted gateway's tool calls actually recover. Not justified as shipped
What this change shipsInventory (7 items) — 6 justifiedIntent: FIX — pooled-topology tool calls are all refused because the tool-policy read goes out without the session token the dispatch already holds (#12441, red-before-green tests).
I verified the load-bearing claims in the checkout: [FIRST-PRINCIPLES-REVIEWED] ed15d5f |
Opus 4.8 Review — ✅ no blocking findingsReviewed Review detailsNo candidates to falsify, and my own verification confirms the change is fail-closed: reading the error body happens once, the No findings. [OPUS-REVIEWED] ed15d5f Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
|
buluoray
left a comment
There was a problem hiding this comment.
Approving. I verified the ordering premise, the call-site coverage, and the fail-closed direction against source rather than from the docstrings, then mutation-verified the new guards. I also have the real-host confirmation this PR says it could not get.
Real-host confirmation on a pooled gateway
The PR notes that "confirming the fix on that host requires updating and restarting its gateway". I have a host in exactly that state, so here is that confirmation.
First, that it is the right code and the right configuration — a green result on a non-pooled host would prove nothing, since a non-pooled backend has the env token and never reaches this bug:
mcp_gateway.enabled=True,gatewayd --max-backends 64, backend stubs carrying--poolable. Pooling on.- The on-disk
src/kiro_crew/mcp_shared.pyserving the gateway hashes to7254fad56b9e97183853addfe115f7d42a48bf1c, byte-identical to this PR's head blob. All three changed files match.
Before the restart, on the same host, every kirocrew-core call was refused — reads included, not only writes:
Refusing tool call 'artifact_list': session dashboard:chat-263-... tool policy unresolved (policy_unreadable)
Asking the endpoint directly returned the 409 this PR narrows:
{"error": "The execution identity is unavailable; Global memory was not used.",
"code": "member_identity_unavailable"}and the pooled backend's environment carried no KIROCREW_STUB_SESSION_TOKEN, matching session_token_header's note that a pooled backend "has no token of its own there".
After the restart, on this code:
| check | result |
|---|---|
artifact_list (read) |
succeeds |
artifact_save (the originally reported failure) |
succeeds |
refusals in the post-restart gateway.log |
0 |
One correction to my own evidence, in case it saves someone a wrong conclusion: grep policy_unreadable audit.log returns hits on this host, but they are my own probe commands containing that string, not refusal events. The load-bearing evidence is the zero count in the fresh gateway.log plus the calls actually succeeding.
What I checked in source
The ordering premise is real, on both dispatch paths. This is the whole basis of the change, so I read it rather than trusting the docstring: _caller_tool_policy is called at mcp_shared.py:1329 and :1619, while set_current_caller runs at :1362 and :1706 — after each. current_caller() is genuinely empty at policy-check time in both, so the ContextVar could not have supplied the token.
Call-site coverage is complete. There are three references to _resolve_tool_policy: the definition, :1314 inside _caller_tool_policy (threads the token), and :900 inside _resolve_excluded_tools, which does not. I checked whether that third one is a gap and it is not — _resolve_excluded_tools has no production caller (only tests, plus a docstring mention in session_token_sig.py:436), and its own docstring scopes it to diagnostics that carry "no member-memory capability or proof". It keeps the ContextVar fallback, which is correct for a caller that has one installed.
No double-read of the error body. _http_error_code consumes exc.read(), and an HTTPError body reads once. The only other .read() in the module is :693 on the success response, a different object, and no arm downstream of the 409 branch touches the body. So the new parse cannot starve an existing reader.
Fail-closed is not relaxed. identity_unattested joins _UNRESOLVED_REFUSES_CALL, so the new reason refuses exactly as policy_unreadable did; only the reason and the operator-facing text change. The narrowing is also one-directional — an unknown, non-JSON, non-object, or non-string-code body keeps the historical policy_unreadable arm, which the parametrized test pins across all four shapes.
Mutation verification
Run in a detached worktree at this PR's head, never against the live tree, so a pooled backend spawning mid-run could not load mutated code. Baseline 135 passed.
| mutation | result |
|---|---|
_tok = caller_token → _tok = "" (pre-fix behavior) |
2 red: test_explicit_caller_token_rides_the_read_with_no_caller_var_set, test_an_explicit_token_outranks_the_caller_var |
_http_error_code returns "" unconditionally |
1 red: test_409_member_identity_unavailable_is_identity_unattested |
drop identity_unattested from _UNRESOLVED_REFUSES_CALL |
2 red, incl. test_identity_unattested_refuses_with_the_identity_message |
_caller_tool_policy stops passing the token |
1 red: test_tools_call_hands_the_caller_block_token_to_the_policy_read |
The third is the one I would point at. Removing the reason from the refusing set flipped the dispatch-level assertion from ran == [] to ran == ['echo'] — the tool actually executed. That test pins fail-closed at the enforcement seam, not merely set membership, which is the difference between a guard and a tautology.
One note, not a request for a change
_resolve_excluded_tools is now the only path that reads a policy without a token argument. It is unreachable from production today, so this is not a defect here. But it is the shape that would silently reintroduce this bug if some future caller picks it up on a pooled backend — worth a line in its docstring, or its removal, whenever someone is next in that file.
…ad (kirodotdev#12445) The tools/call dispatch checks the session's tool policy on the read loop, before the worker installs the caller ContextVar. The policy resolver read the per-session token from that ContextVar, so at the moment it mattered the token was always empty and the loopback request went out with X-Session-Key and no X-Session-Token. A pooled control-plane backend has the token nowhere else: gatewayd spawns it from the daemon's environment and hands the token over per frame in the caller block. Since the gateway began requiring an attestation behind a declared session key, every such read was answered 409 member_identity_unavailable, the client read every 409 as policy_unreadable, and every kirocrew-core / kirocrew-cron tool call on a pooled host was refused with a message pointing at the agents directory. _resolve_tool_policy now takes the caller's token explicitly, handed in by _caller_tool_policy from the CallerContext it already receives; the ContextVar stays as the fallback for callers that arrive with it set. The 409 arm reads the body's code: member_identity_unavailable becomes its own reason, identity_unattested, with its own audit event and an error text that names the missing token. It stays in the refusing set, so nothing an operator excludes becomes unenforced; only the diagnosis changes. The unreadable-spec refusal keeps its reason and drops the misleading "Retry shortly". Fixes kirodotdev#12441 Co-authored-by: Nick Bowers <nrb@amazon.com>
Problem / Motivation
On a host running the pooled MCP topology, every
kirocrew-coreandkirocrew-crontool call is refused once the gateway includes #12164:29 of 29 recorded calls on one host in the 3.5 hours after its gateway restart:
spawn_run,send_message,cron_add,memory_recall,artifact_save, the ledger tools, all of them, across dashboard, Slack and cron sessions. "Retry shortly" is wrong -- it never recovers -- and the reason points at the agents directory, where nothing is wrong.Why it matters
The whole Kiro Crew control plane is dark on that host: no subagents, no crons that call tools, no notifications, no memory. And the message sends the operator to fix agent specs (the #12312 diagnosis) when the real cause is that this process never sent its own credential. Nothing in the logs on the client side says so.
What changed (motivation → approach → change)
caller_token=); the ContextVar stays as the fallbackpolicy_unreadablecodedecides:member_identity_unavailable->identity_unattested; anything else ->policy_unreadableas beforeidentity_unattestednames the missing or mismatched session token;policy_unreadablenames the unreadable specThe gateway now requires a declared
X-Session-Keyto come with an attestation (#12164).mcp_shared._resolve_tool_policysent the token fromcurrent_caller(). Buttools/callchecks the policy on the read loop, andset_current_callerruns later inside the worker, so at the checkcurrent_caller()isNoneand the token is"". The env fallback is empty too: a pooled control-plane backend is spawned from gatewayd's environment and receives the token per frame in the caller block (gatewayd._caller_for_backend). So the read went out with a key and no token, the gateway answered 409member_identity_unavailable, and the client mapped every 409 topolicy_unreadable, the one reason that refuses a call.The fix hands
_caller_ctx.session_token-- the object the check already has -- into the resolver. Non-pooled stdio servers were never affected (their environment carries the token), which is why the repository's end-to-end lane did not see this.identity_unattestedstays in_UNRESOLVED_REFUSES_CALL. It is the same withheld-deny argument aspolicy_unreadable: the spec was never read, so what it excludes is unknown. No exclusion enforced today becomes unenforced; only the reason and the message change. A 409 whose body is missing, unparseable or carries any othercodekeeps the status's historical meaning.Backwards compatibility
Compatible:
_resolve_tool_policygains a keyword-onlycaller_token=""; every existing caller is unchanged._UNRESOLVED_REFUSES_CALLgains a reason that previously surfaced aspolicy_unreadable, so no call that ran before is refused now, and the one that was refused is still refused with a truthful message.Tests
test_mcp_shared.py::TestStdioLoopCallerIdentity::test_tools_call_hands_the_caller_block_token_to_the_policy_read-- the dispatch passes the caller block's token to the resolver whilecurrent_caller()is stillNone, and a block without a token passes"", never a stale one.test_mcp_shared.py::...::test_identity_unattested_refuses_with_the_identity_message-- the call does not run, the text names the token and not the agents directory, the audit error ismanagedToolPolicy.unresolved:identity_unattested.test_mcp_shared_cache.py::TestCallerTokenOnTheRead(4) -- explicit token rides the request with no ContextVar set; the ContextVar is still the fallback; an explicit token outranks it; no token anywhere sends no header.test_mcp_shared_cache.py::TestLongCacheFailures::test_409_member_identity_unavailable_is_identity_unattested-- own reason, own audit event (tool_policy.unattested,token=absent), in the refusing set, never negative-cached....::test_409_spec_body_or_unreadable_body_stays_policy_unreadable(4 cases) -- spec code, garbage, non-object, non-string code all staypolicy_unreadable.Red-before-green: with
mcp_shared.pyatorigin/mainand the new tests in place, 5 of the 7 new cases fail (TypeError: unexpected keyword argument 'caller_token';'policy_unreadable' == 'identity_unattested'); the two that pass pin fallback behaviour the fix does not touch.Manual verification
Diagnosed on a live pooled host: 67 gateway
api_session_tool_policydenials "The execution identity is unavailable." with zero unreadable-spec denials and zero signature-mismatch warnings, the first denial in the same millisecond as the first client refusal. Confirming the fix on that host requires updating and restarting its gateway; the unit tests pin the wire-level behaviour (header present, 409 body decoded) that the live check would exercise.Related Issues
Fixes #12441
Pattern harvest
Rule candidate: review-prompt
Pattern: a per-call value read from a ContextVar/thread-local on a code path that runs before the dispatch installs it -- check that every reader of
current_caller()executes afterset_current_caller()on the same path.Checklist
feat|fix|docs|refactor|perf|test|chore|ci|build|revert: ...)