Skip to content

Mark a callback's tool answer as refused only when it was refused, as the in-process door does - #570

Merged
davidmckayv merged 2 commits into
CopilotKit:mainfrom
kevin9327:mark-only-refusals-on-callback
Sep 15, 2026
Merged

davidmckayv merged 2 commits into
CopilotKit:mainfrom
kevin9327:mark-only-refusals-on-callback

Conversation

@kevin9327

Copy link
Copy Markdown
Contributor

What this changes

/api/agent-tools/call is the door a Bot running its own loop (the LangGraph Bots) calls tools back through. It put REFUSAL_MARKER in front of every throw from pluginStore.callTool. callTool throws for three different reasons:

  • a boundary holding: PluginRefusedError
  • a vendor that broke: rethrown unchanged after callTool records mcp.call_failed (store.ts, the catch that ends in throw error)
  • a fault of this deployment's own, for example a query failure

The marker is not just wording. chat-transcript.tsx checks answer.startsWith(REFUSAL_MARKER) and labels that tool line as blocked, and the model reads "Refused." as "not allowed". So on these Bots:

  • a vendor outage reached the person as a blocked call and the model as Refused. fetch failed
  • a database fault reached them as Refused. That tool could not be called.

Meanwhile the audit trail recorded a failed call. The route's own test says matching the in-process door on a fault "is the property". The in-process door (grantedTools) has kept the three apart since #31: "A vendor that failed is not a refusal… one means 'not allowed', the other means 'it broke'."

The route now gives the in-process door's three answers:

Throw Before After
PluginRefusedError Refused. <reason> Refused. <reason> (unchanged)
a vendor's Error Refused. <sentence> That tool could not be called: <sentence>
a deployment fault, or a non-Error Refused. That tool could not be called. That tool could not be called.

The route keeps what it already had over the in-process door. Everything still goes through withoutStatement, and the deployment-fault shelf is unchanged, so nothing new is relayed. isError stays true on all three.

For completeness, the host tools behind deploymentToolCaller share this catch, but they are not affected. Their answer helper already catches its own failures and marks only a HostAccessRefusedError, so they never throw here.

Where it runs

  • New state that outlives a request? None.
  • What happens on the second replica? The same answer on every replica.
  • Anything serialised? No.
  • Anything fanned out to a browser? No new fan-out. The transcript draws the same answer it already receives.
  • New listener, port, or schedule? No.

Boundary and audit

  • Every acting call still goes through the gateway: resolve, decide, audit, then act. callTool is untouched.
  • New refusals and new failures each write a row. None are added. callTool already writes mcp.call_refused or mcp.call_failed before it throws, and this makes the transcript agree with that row.
  • Nothing new is trusted from the client that the server can resolve itself.

Changelog

  • A line in CHANGELOG.md under Unreleased.

Overlap with #569, which also touches this route: the two merge cleanly except on the CHANGELOG.md anchor and two adjacent import lines in app.ts. #569 adds vendorAnswer to the ./plugins/tools import, and this PR adds PluginRefusedError to the ./plugins/store import just above it. Resolving is keeping both, and the test file merges on its own. I checked with git merge-tree. I'm happy to rebase whichever lands second.

Proof

New test in server/tests/agent-callback-token.test.ts, in the route's existing describe: "a throw is marked as a refusal only when it is one, the way the in-process door marks it". For each of the three throws, it asks the route (the file's existing toolResult harness, with a signed run) and grantedTools(...)[0].execute({}) about the same throwing store. It records whether the route's answer carries the marker and whether it matches the in-process answer.

On main:

error: expect(received).toEqual(expected)

  [
-   "vendor failure: marked false, same as in-process true",
-   "deployment fault: marked false, same as in-process true",
+   "vendor failure: marked true, same as in-process false",
+   "deployment fault: marked true, same as in-process false",
    "refusal: marked true, same as in-process true",

(fail) the tool-call route a callback token guards > a throw is marked as a refusal only when it is one, the way the in-process door marks it

With the fix:

  • bun test tests/agent-callback-token.test.ts tests/host-access-callback-route.test.ts: 32 pass, 0 fail. These include the existing route tests: a query failure never reaches the model, a refusal is still relayed in full, and a thrown vendor sentence is still relayed.
  • bun run typecheck (app, server, worker): exit 0.
  • bunx biome check on the changed files: clean.

All three arms of the new catch run in the new test. Neither LangGraph Bot looks for a leading "Refused." in what the deployment returns; they only write their own refusals, so nothing downstream keys on the old marker.

🤖 Generated with Claude Code

… the in-process door does

`/api/agent-tools/call` put `REFUSAL_MARKER` in front of every throw
from `callTool`. `callTool` throws `PluginRefusedError` when a boundary
holds, and rethrows a vendor that broke after recording
`mcp.call_failed`. A database fault can throw from it too.

The marker is what the transcript draws: `chat-transcript.tsx` labels a
tool result that starts with it as blocked, and the model reads
"Refused." as "not allowed". So for a Bot calling tools back from its
own process, a vendor outage ("Refused. fetch failed") or a fault of
this deployment's own ("Refused. That tool could not be called.") was
shown to the person as a policy refusing, while the audit trail said the
call had failed.

The in-process door (`grantedTools`) has kept these apart from the
start: a refusal is marked, a vendor that failed reads "That tool could
not be called: <sentence>", and a deployment fault reads "That tool
could not be called." The route now gives the same three answers. The
refusal's text is unchanged, and the route still passes everything
through `withoutStatement` and keeps the deployment-fault shelf.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

@davidmckayv

Copy link
Copy Markdown
Contributor

Thanks for these — all three are clean and going in. One small process nudge, not a criticism: #569 and #570 are really two halves of the same idea (make the callback door behave like the in-process door — one for the resolved-error path, one for the thrown path), on the same handler and the same test file. When fixes are that tightly coupled, feel free to land them as a single PR — it's a little less review overhead and skips the rebase between them. The work itself is careful and well-tested; keep it coming.

# Conflicts:
#	CHANGELOG.md
#	server/src/app.ts
@davidmckayv
davidmckayv merged commit 93bb0d1 into CopilotKit:main Sep 15, 2026
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants