Skip to content

fix(mcp): send the caller block's session token on the tool-policy read - #12445

Merged
NicholasRBowers merged 1 commit into
mainfrom
fix/tool-policy-caller-token
Sep 21, 2026
Merged

NicholasRBowers merged 1 commit into
mainfrom
fix/tool-policy-caller-token

Conversation

@NicholasRBowers

Copy link
Copy Markdown
Contributor

Problem / Motivation

On a host running the pooled MCP topology, every kirocrew-core and kirocrew-cron tool call is refused once the gateway includes #12164:

tool 'X' is unavailable because this session's tool policy could not be read (policy_unreadable); refusing the call rather than ignoring an operator's exclusion list. Retry shortly.

29 of 29 recorded calls on one host in the 3.5 hours after its gateway restart: spawn_run, send_message, cron_add, memory_recall, artifact_save, the ledger tools, all of them, across dashboard, Slack and cron sessions. "Retry shortly" is wrong -- it never recovers -- and the reason points at the agents directory, where nothing is wrong.

Why it matters

The whole Kiro Crew control plane is dark on that host: no subagents, no crons that call tools, no notifications, no memory. And the message sends the operator to fix agent specs (the #12312 diagnosis) when the real cause is that this process never sent its own credential. Nothing in the logs on the client side says so.

What changed (motivation → approach → change)

Piece Before After
token on the tool-policy read read from the caller ContextVar, which the dispatch has not installed yet at that point, so always empty handed in explicitly from the caller block the dispatch already holds (caller_token=); the ContextVar stays as the fallback
a 409 from the gateway always policy_unreadable body code decides: member_identity_unavailable -> identity_unattested; anything else -> policy_unreadable as before
the refusal text one message for both, ending "Retry shortly" identity_unattested names the missing or mismatched session token; policy_unreadable names the unreadable spec

The gateway now requires a declared X-Session-Key to come with an attestation (#12164). mcp_shared._resolve_tool_policy sent the token from current_caller(). But tools/call checks the policy on the read loop, and set_current_caller runs later inside the worker, so at the check current_caller() is None and the token is "". The env fallback is empty too: a pooled control-plane backend is spawned from gatewayd's environment and receives the token per frame in the caller block (gatewayd._caller_for_backend). So the read went out with a key and no token, the gateway answered 409 member_identity_unavailable, and the client mapped every 409 to policy_unreadable, the one reason that refuses a call.

The fix hands _caller_ctx.session_token -- the object the check already has -- into the resolver. Non-pooled stdio servers were never affected (their environment carries the token), which is why the repository's end-to-end lane did not see this.

identity_unattested stays in _UNRESOLVED_REFUSES_CALL. It is the same withheld-deny argument as policy_unreadable: the spec was never read, so what it excludes is unknown. No exclusion enforced today becomes unenforced; only the reason and the message change. A 409 whose body is missing, unparseable or carries any other code keeps the status's historical meaning.

Backwards compatibility

Compatible: _resolve_tool_policy gains a keyword-only caller_token=""; every existing caller is unchanged. _UNRESOLVED_REFUSES_CALL gains a reason that previously surfaced as policy_unreadable, so no call that ran before is refused now, and the one that was refused is still refused with a truthful message.

Tests

  • test_mcp_shared.py::TestStdioLoopCallerIdentity::test_tools_call_hands_the_caller_block_token_to_the_policy_read -- the dispatch passes the caller block's token to the resolver while current_caller() is still None, and a block without a token passes "", never a stale one.
  • test_mcp_shared.py::...::test_identity_unattested_refuses_with_the_identity_message -- the call does not run, the text names the token and not the agents directory, the audit error is managedToolPolicy.unresolved:identity_unattested.
  • test_mcp_shared_cache.py::TestCallerTokenOnTheRead (4) -- explicit token rides the request with no ContextVar set; the ContextVar is still the fallback; an explicit token outranks it; no token anywhere sends no header.
  • test_mcp_shared_cache.py::TestLongCacheFailures::test_409_member_identity_unavailable_is_identity_unattested -- own reason, own audit event (tool_policy.unattested, token=absent), in the refusing set, never negative-cached.
  • ...::test_409_spec_body_or_unreadable_body_stays_policy_unreadable (4 cases) -- spec code, garbage, non-object, non-string code all stay policy_unreadable.

Red-before-green: with mcp_shared.py at origin/main and the new tests in place, 5 of the 7 new cases fail (TypeError: unexpected keyword argument 'caller_token'; 'policy_unreadable' == 'identity_unattested'); the two that pass pin fallback behaviour the fix does not touch.

Manual verification

Diagnosed on a live pooled host: 67 gateway api_session_tool_policy denials "The execution identity is unavailable." with zero unreadable-spec denials and zero signature-mismatch warnings, the first denial in the same millisecond as the first client refusal. Confirming the fix on that host requires updating and restarting its gateway; the unit tests pin the wire-level behaviour (header present, 409 body decoded) that the live check would exercise.

Related Issues

Fixes #12441

Pattern harvest

Rule candidate: review-prompt
Pattern: a per-call value read from a ContextVar/thread-local on a code path that runs before the dispatch installs it -- check that every reader of current_caller() executes after set_current_caller() on the same path.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable) -- N/A, module docstrings and comments carry the contract
  • No secrets, credentials, or internal references in the diff

The tools/call dispatch checks the session's tool policy on the read
loop, before the worker installs the caller ContextVar. The policy
resolver read the per-session token from that ContextVar, so at the
moment it mattered the token was always empty and the loopback request
went out with X-Session-Key and no X-Session-Token. A pooled
control-plane backend has the token nowhere else: gatewayd spawns it
from the daemon's environment and hands the token over per frame in the
caller block. Since the gateway began requiring an attestation behind a
declared session key, every such read was answered 409
member_identity_unavailable, the client read every 409 as
policy_unreadable, and every kirocrew-core / kirocrew-cron tool call on
a pooled host was refused with a message pointing at the agents
directory.

_resolve_tool_policy now takes the caller's token explicitly, handed in
by _caller_tool_policy from the CallerContext it already receives; the
ContextVar stays as the fallback for callers that arrive with it set.
The 409 arm reads the body's code: member_identity_unavailable becomes
its own reason, identity_unattested, with its own audit event and an
error text that names the missing token. It stays in the refusing set,
so nothing an operator excludes becomes unenforced; only the diagnosis
changes. The unreadable-spec refusal keeps its reason and drops the
misleading "Retry shortly".

Fixes #12441
@NicholasRBowers
NicholasRBowers requested review from a team and iamwhatever September 21, 2026 04:17
@NicholasRBowers

Copy link
Copy Markdown
Contributor Author

Intent: A pooled control-plane MCP backend proves which session it acts for when it reads that session's tool policy, so the gateway's attestation gate (#12164) no longer refuses every tool call on a pooled host; and when the gateway does refuse for want of that proof, the refusal says so instead of blaming an unreadable agent spec.
Not a goal: Changing which unresolved reasons refuse a call (identity_unattested is fail-closed like policy_unreadable), relaxing the gateway's attestation requirement, or fixing the separate unreadable-spec producer in #12312.

@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Sep 21, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — ✅ PASS

Design-level review of ed15d5f7913f96285d83bab0c576f5391b011908 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Verify the deferred live check: on the diagnosed pooled host, after gateway restart, tool calls recover and the audit shows no tool_policy.unattested events.

[DESIGN-REVIEWED] ed15d5f

@github-actions

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ✅ no blocking findings

GPT 5.6 completed its review of ed15d5f7913f96285d83bab0c576f5391b011908 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] ed15d5f

False positive or not applicable? A repository writer can comment:
/ai-review override gpt ed15d5f7913f96285d83bab0c576f5391b011908: <one-sentence reason>

@github-actions

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5) — ✅ PASS

Premise-level review of ed15d5f7913f96285d83bab0c576f5391b011908 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

The fix is pinned only by unit tests — confirm on the live pooled host that a restarted gateway's tool calls actually recover.

Not justified as shipped

  • Item 4 — rides along: the policy_unreadable message rewrite ("fix or remove the unreadable spec") is not the reported defect, which was the identity case; the "Retry shortly" harm still names it, so nothing more.

What this change ships

Inventory (7 items) — 6 justified

Intent: FIX — pooled-topology tool calls are all refused because the tool-policy read goes out without the session token the dispatch already holds (#12441, red-before-green tests).

  1. Pooled backends' tool calls work again: the policy read sends the caller block's token — justified
  2. An unattested read now refuses as identity_unattested, naming the missing token instead of the agents directory — justified
  3. identity_unattested joins the fail-closed refusing set (same calls refused as before, truthful reason) — justified
  4. The unreadable-spec refusal drops "Retry shortly" and names the spec remedy — rides along: declared relabel, not the reported defect
  5. New SEL audit event tool_policy.unattested with token presence — justified
  6. Unknown or garbled 409 bodies keep the historical policy_unreadable meaning — justified
  7. _resolve_tool_policy gains keyword-only caller_token with the ContextVar as fallback — justified

I verified the load-bearing claims in the checkout: _caller_tool_policy runs on the read loop (src/kiro_crew/mcp_shared.py:1314) while set_current_caller runs in the worker (line 1362); the gateway does emit 409 bodies with "code": "member_identity_unavailable" (4 sites in dashboard/handlers/); the only "Retry shortly" test pin is memory-startup's, untouched; other current_caller() readers (mcp_core, mcp_cron) run in the worker after installation — 0 unfixed siblings of the ContextVar-read-before-install cause.

[FIRST-PRINCIPLES-REVIEWED] ed15d5f

@github-actions

Copy link
Copy Markdown
Contributor

Opus 4.8 Review — ✅ no blocking findings

Reviewed ed15d5f7913f96285d83bab0c576f5391b011908 — this comment is updated in place on each push.

Review details

No candidates to falsify, and my own verification confirms the change is fail-closed: reading the error body happens once, the policy_unreadable arm never re-reads it, session_token originates only from the caller block, and any unattested/foreign token yields identity_unattested which stays in _UNRESOLVED_REFUSES_CALL (deny-by-default). Nothing widens a permission. Nothing survives to report.

No findings.

[OPUS-REVIEWED] ed15d5f

Verdict parsed from the review's SHA-scoped output markers for commit ed15d5f7913f96285d83bab0c576f5391b011908.

False positive or not applicable? A repository writer can comment:
/ai-review override fable ed15d5f7913f96285d83bab0c576f5391b011908: <one-sentence reason>

@NicholasRBowers

Copy link
Copy Markdown
Contributor Author
  • Item 4 — policy_unreadable message rewrite rides along span=2fb1e8fe8523

rebutted: the reported defect is that two different 409 refusals shared one message, so a reader of either could not tell "no token reached this call" from "the operator's spec is malformed". Splitting the reason without splitting the text would leave the spec arm still describing the identity arm's symptom ("Retry shortly"), which #12312 independently reports as wrong for the spec case too.
The spec-arm text changes only the sentence after the colon and the trailing advice; its reason, audit event and fail-closed disposition are byte-identical to main. test_unresolved_policy_refuses_the_call pins the unchanged part ("tool policy could not be read" / policy_unreadable) and still passes.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Sep 21, 2026
@NicholasRBowers
NicholasRBowers enabled auto-merge (squash) September 21, 2026 04:39

@buluoray buluoray left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving. I verified the ordering premise, the call-site coverage, and the fail-closed direction against source rather than from the docstrings, then mutation-verified the new guards. I also have the real-host confirmation this PR says it could not get.

Real-host confirmation on a pooled gateway

The PR notes that "confirming the fix on that host requires updating and restarting its gateway". I have a host in exactly that state, so here is that confirmation.

First, that it is the right code and the right configuration — a green result on a non-pooled host would prove nothing, since a non-pooled backend has the env token and never reaches this bug:

  • mcp_gateway.enabled=True, gatewayd --max-backends 64, backend stubs carrying --poolable. Pooling on.
  • The on-disk src/kiro_crew/mcp_shared.py serving the gateway hashes to 7254fad56b9e97183853addfe115f7d42a48bf1c, byte-identical to this PR's head blob. All three changed files match.

Before the restart, on the same host, every kirocrew-core call was refused — reads included, not only writes:

Refusing tool call 'artifact_list': session dashboard:chat-263-... tool policy unresolved (policy_unreadable)

Asking the endpoint directly returned the 409 this PR narrows:

{"error": "The execution identity is unavailable; Global memory was not used.",
 "code": "member_identity_unavailable"}

and the pooled backend's environment carried no KIROCREW_STUB_SESSION_TOKEN, matching session_token_header's note that a pooled backend "has no token of its own there".

After the restart, on this code:

check result
artifact_list (read) succeeds
artifact_save (the originally reported failure) succeeds
refusals in the post-restart gateway.log 0

One correction to my own evidence, in case it saves someone a wrong conclusion: grep policy_unreadable audit.log returns hits on this host, but they are my own probe commands containing that string, not refusal events. The load-bearing evidence is the zero count in the fresh gateway.log plus the calls actually succeeding.

What I checked in source

The ordering premise is real, on both dispatch paths. This is the whole basis of the change, so I read it rather than trusting the docstring: _caller_tool_policy is called at mcp_shared.py:1329 and :1619, while set_current_caller runs at :1362 and :1706 — after each. current_caller() is genuinely empty at policy-check time in both, so the ContextVar could not have supplied the token.

Call-site coverage is complete. There are three references to _resolve_tool_policy: the definition, :1314 inside _caller_tool_policy (threads the token), and :900 inside _resolve_excluded_tools, which does not. I checked whether that third one is a gap and it is not — _resolve_excluded_tools has no production caller (only tests, plus a docstring mention in session_token_sig.py:436), and its own docstring scopes it to diagnostics that carry "no member-memory capability or proof". It keeps the ContextVar fallback, which is correct for a caller that has one installed.

No double-read of the error body. _http_error_code consumes exc.read(), and an HTTPError body reads once. The only other .read() in the module is :693 on the success response, a different object, and no arm downstream of the 409 branch touches the body. So the new parse cannot starve an existing reader.

Fail-closed is not relaxed. identity_unattested joins _UNRESOLVED_REFUSES_CALL, so the new reason refuses exactly as policy_unreadable did; only the reason and the operator-facing text change. The narrowing is also one-directional — an unknown, non-JSON, non-object, or non-string-code body keeps the historical policy_unreadable arm, which the parametrized test pins across all four shapes.

Mutation verification

Run in a detached worktree at this PR's head, never against the live tree, so a pooled backend spawning mid-run could not load mutated code. Baseline 135 passed.

mutation result
_tok = caller_token → _tok = "" (pre-fix behavior) 2 red: test_explicit_caller_token_rides_the_read_with_no_caller_var_set, test_an_explicit_token_outranks_the_caller_var
_http_error_code returns "" unconditionally 1 red: test_409_member_identity_unavailable_is_identity_unattested
drop identity_unattested from _UNRESOLVED_REFUSES_CALL 2 red, incl. test_identity_unattested_refuses_with_the_identity_message
_caller_tool_policy stops passing the token 1 red: test_tools_call_hands_the_caller_block_token_to_the_policy_read

The third is the one I would point at. Removing the reason from the refusing set flipped the dispatch-level assertion from ran == [] to ran == ['echo'] — the tool actually executed. That test pins fail-closed at the enforcement seam, not merely set membership, which is the difference between a guard and a tautology.

One note, not a request for a change

_resolve_excluded_tools is now the only path that reads a policy without a token argument. It is unreachable from production today, so this is not a defect here. But it is the shape that would silently reintroduce this bug if some future caller picks it up on a pooled backend — worth a line in its docstring, or its removal, whenever someone is next in that file.

@NicholasRBowers
NicholasRBowers merged commit 9e7ad91 into main Sep 21, 2026
92 checks passed
@NicholasRBowers
NicholasRBowers deleted the fix/tool-policy-caller-token branch September 21, 2026 05:04
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Sep 21, 2026
amadsalmon pushed a commit to amadsalmon/KiroCrew that referenced this pull request Sep 22, 2026
…ad (kirodotdev#12445)

The tools/call dispatch checks the session's tool policy on the read
loop, before the worker installs the caller ContextVar. The policy
resolver read the per-session token from that ContextVar, so at the
moment it mattered the token was always empty and the loopback request
went out with X-Session-Key and no X-Session-Token. A pooled
control-plane backend has the token nowhere else: gatewayd spawns it
from the daemon's environment and hands the token over per frame in the
caller block. Since the gateway began requiring an attestation behind a
declared session key, every such read was answered 409
member_identity_unavailable, the client read every 409 as
policy_unreadable, and every kirocrew-core / kirocrew-cron tool call on
a pooled host was refused with a message pointing at the agents
directory.

_resolve_tool_policy now takes the caller's token explicitly, handed in
by _caller_tool_policy from the CallerContext it already receives; the
ContextVar stays as the fallback for callers that arrive with it set.
The 409 arm reads the body's code: member_identity_unavailable becomes
its own reason, identity_unattested, with its own audit event and an
error text that names the missing token. It stays in the refusing set,
so nothing an operator excludes becomes unenforced; only the diagnosis
changes. The unreadable-spec refusal keeps its reason and drops the
misleading "Retry shortly".

Fixes kirodotdev#12441

Co-authored-by: Nick Bowers <nrb@amazon.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Pooled backends send no X-Session-Token on the tool-policy read; every tool call refused as policy_unreadable

2 participants