feat(mcp): classify tool errors on spans and give agents actionable, masked error text by calebevans-ab · Pull Request #1355 · airbytehq/PyAirbyte

Caleb Evans (calebevans-ab) · 2026-10-09T21:17:06Z

TL;DR: When a tool call fails, its span in Datadog and OTel now says what kind of failure it was and where in the code it happened, without exporting exception text or IDs. Agents get a fixed message they can act on instead of a debug dump, and argument values echoed in that message are masked.

When a tool call fails, the prod mcp.tools_call span in Datadog only shows a class name and "tool resulted in an error". Agents get API error occurred: API error occurred plus the workspace ID, API root and request URL. This includes the changes from #1354 (Cam's draft), merged here.

What a failed span gains

Tag (`airbyte.mcp.` prefix)	Example for an upstream 500
`error.category`	`upstream_error`
`error.fault`	`upstream`
`upstream.status_code` (numeric)	`500`
`error.cause_types`	`["SDKError"]`
`error.reason`	`unexpected-problem`

error.message: becomes tool resulted in an error (upstream_error, HTTP 500, unexpected-problem).
Frames-only stack: failed spans also get a stack that records only the code location, one module:qualname:lineno line per frame:
```
AirbyteCloudError
  fastmcp.server.server:FastMCP.call_tool:1517
  ...
caused by SDKError
  airbyte._util.api_util:...
```
- On Datadog it's error.stack, which also feeds the error panel and LLM Obs. On OTel it's airbyte.mcp.error.stack / exception.stacktrace, built by fastmcp-extensions#148.
- Runs of 3+ framework frames (fastmcp, anyio, pydantic, ...) before our code collapse to first / ... / last. A run at the raise site is kept whole.
Never exported: exception messages, args, locals, source lines, file paths, response details or IDs.

Changes

Tracing

Native Datadog backend: failed spans get the tags above, from the same fastmcp-extensions classifier the OTel backends use. Returned error results are classified too. The exception is still never passed to ddtrace; the stack comes from fastmcp_extensions error_stack.
error.reason: the Airbyte API problem slug from the error response body. It's reported on both backends through the library's new error_reason hook. (It was error.problem_type in feat(mcp): classify tool errors on native Datadog spans #1354.)
- The slug comes from type, or from title only when type is the generic errors URL. A missing or non-string type gives no reason.
- When the raw-HTTP helpers have already parsed the slug into error context, that validated slug is used.
Invalid arguments and unknown tools: classified on the Datadog span the same way as on the OTel backends (invalid_arguments / caller, unknown_tool / caller).
datadog-otlp backend: the LLM Obs metadata exporter keeps category, fault, reason, cause types and status code.
Sync jobs: a failed or timed-out sync reports as upstream_error / upstream_timeout on both backends.
Bug fix: get_source_definition and get_destination_definition now wrap SDKError, so a 404 no longer surfaces as a raw SDKError.

Agent-facing error text

Cloud errors: describe_cloud_error(parse_cloud_error(status, body)) in the new airbyte/_util/cloud_errors.py maps Cloud's problem type to a fixed message plus one recovery step.
- The table is in airbyte/_util/cloud_errors.yaml; types not in it fall back to a status-based message.
- Maintenance rules for the table are in its header, with a short section in CONTRIBUTING.md and a pointer in AGENTS.md.
- Cloud's detail and data.message are never relayed; on a 5xx, data.message can be raw SQL. data.resourceType is shown only if it's in a fixed list (source, connection, ...).
- Direct-access errors (AirbyteCloudApiError) use the same table, keeping their fixed 401/403/gateway advice, so the request URL and response body never reach the agent.
- Examples:
```
500: Airbyte Cloud hit an unexpected error. This is often caused by an invalid argument value. (Cloud error: unexpected-problem, HTTP 500) Check the arguments (ranges, IDs, formats). If they look right, don't retry more than once; tell the user.
401: Airbyte Cloud rejected the credentials. (HTTP 401) Ask the user to check the client ID/secret or token; don't retry.
```
New AgentErrorTextMiddleware: sits outside UserFacingErrorMiddleware and telemetry, so classification is unchanged. It:
- drops the context dump;
- rebuilds argument-validation errors without pydantic's input_value;
- masks echoed argument values as <value of source_id>, naming only the top-level parameter, never a nested dict key.
How masking decides: arguments are classified with the same rules as arg tracing.
- Secret-like arguments are masked wherever they appear.
- Other arguments are masked as whole tokens of 8 or more characters, so words like user in the guidance stay intact. Secrets shorter than 4 characters are also masked only as whole tokens.
- A list or dict passed to an enum argument is masked instead of crashing the middleware.
Raw-HTTP helpers: _get_public_api_json and _make_config_api_request no longer put the request body or the response object in error context, which could hold connector secrets. This also changes str(error) for library users.
Config validation: errors name the JSON path and the failed rule instead of quoting the value.
Local MCP tools: they no longer return tracebacks to the agent. They log only the exception class and a frames-only stack, so hosted JSON logs get no exception text.

Depends on

airbytehq/fastmcp-extensions#148, released in fastmcp-extensions 0.34.1. This PR pins the released fastmcp-extensions>=0.34.2,<1.0.0 (0.34.2 adds only the airbytehq/fastmcp-extensions#149 docs move).

Verified locally

Ran a local server on both the datadog and otel backends with real and stubbed API failures:

Tags and stacks come out as described, and successful calls carry no stack.
No sentinel string, exception message, file path, response detail or ID appears in any exported span.

Heads-up

Monitors: ones matching the exact string tool resulted in an error should match it as a prefix.
Status code: it's a numeric metric, not a string tag. LLM Observability metadata carries it as the string "500".
Unknown tools: they now report outcome=unknown_tool on the Datadog span (was exception), matching the OTel backend.
Tool-filter rejections: tagged tool_unavailable / caller, with error.message tool resulted in an error (tool_unavailable). Any tool-call failure that escapes as a protocol error now gets the same summary.
403s: the API returns 403 for a resource that does not exist, so a mistyped ID shows as status 403 with reason forbidden, category user_error.
Failed syncs on native Datadog: the telemetry event is classified without the sync-job classifier or the reason hook. So for a failed sync the span says upstream_error and the event says unclassified. The OTel backends agree.
ID-like reasons: a reason slug that is all digits or contains a UUID is dropped, so an ID in a problem type or title is never exported.
Agent-facing text changes: the messages for Cloud errors other than 403, and for argument-validation errors, are different. Tests that pinned the old text are updated.

🤖 Generated with Claude Code

Link to Devin session: https://app.devin.ai/sessions/85c491e543a74ab89151fdf7a0ba5d20
Open in Devin Desktop: https://app.devin.ai/desktop/session/85c491e543a74ab89151fdf7a0ba5d20?variant=devin
Requested by: Caleb Evans (@calebevans-ab)

The native Datadog backend recorded only the exception class and a fixed "tool resulted in an error" message, so failed tool calls could not be grouped by cause. Prod showed 109 describe_cloud_connection failures that were all upstream 500s, which was only visible by opening child spans. - Reuse fastmcp-extensions' error_attributes so Datadog spans carry the same error.category, error.fault, and upstream.status_code as the OTel backend. - Add error.problem_type: the Airbyte API problem slug from the response body. Only the type (or title, for the generic errors page) is read, and only a short lowercase slug is exported; detail and data are never read. - Build error.message from those bounded tags only. - Classify failed and timed-out sync jobs as upstream faults instead of internal, on both backends. - Wrap SDKError in get_source_definition / get_destination_definition so describe_cloud_connector raises AirbyteMissingResourceError instead of a raw SDKError.

The native Datadog backend showed only an exception class name on a failed `mcp.tools_call` span. Tag it with the error facts fastmcp-extensions already computes for the `otel` backends and the telemetry event: `airbyte.mcp.error.category`, `airbyte.mcp.error.fault`, `airbyte.mcp.error.cause_types`, and `airbyte.mcp.upstream.status_code`. The values come from the library's shared per-call facts, so the span and the event agree. No exception message is read. Pins fastmcp-extensions to the prerelease built from airbytehq/fastmcp-extensions#148. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

github-code-quality · 2026-10-09T21:21:25Z

Code Coverage Overview

Languages: Python

Python / code-coverage/pytest-no-creds

The overall line coverage in commit 1a7ea03 in the caleb/mcp-datadog-er... branch is 78%. The line coverage in commit d9f652f in the release-candidate/v1 branch is 65%.

Show a line coverage summary of the most impacted files.

File	`release-candidate/v1` d9f652f	`caleb/mcp-datadog-er...` 1a7ea03	+/-
`airbyte/mcp/cloud.py`	52%	83%	+31%
`airbyte/_util/api_util.py`	36%	68%	+32%
`airbyte/mcp/_datadog.py`	0%	38%	+38%
`airbyte/cloud/connectors.py`	34%	75%	+41%
`airbyte/mcp/_user_identity.py`	0%	93%	+93%
`airbyte/_direct...ctors/models.py`	0%	94%	+94%
`airbyte/mcp/_telemetry.py`	0%	94%	+94%
`airbyte/mcp/_sso_auth.py`	0%	96%	+96%
`airbyte/cloud/models.py`	0%	97%	+97%
`airbyte/mcp/int...nc_status_ui.py`	0%	97%	+97%

Python / code-coverage/pytest

The overall line coverage in commit 1a7ea03 in the caleb/mcp-datadog-er... branch is 81%. The line coverage in commit d9f652f in the release-candidate/v1 branch is 71%.

Show a line coverage summary of the most impacted files.

File	`release-candidate/v1` d9f652f	`caleb/mcp-datadog-er...` 1a7ea03	+/-
`airbyte/mcp/cloud.py`	52%	83%	+31%
`airbyte/mcp/_datadog.py`	0%	38%	+38%
`airbyte/mcp/_error_handling.py`	0%	89%	+89%
`airbyte/mcp/int..._registry_ui.py`	0%	92%	+92%
`airbyte/mcp/_user_identity.py`	0%	93%	+93%
`airbyte/_direct...ctors/models.py`	0%	94%	+94%
`airbyte/mcp/_telemetry.py`	0%	94%	+94%
`airbyte/mcp/_sso_auth.py`	0%	96%	+96%
`airbyte/cloud/models.py`	0%	97%	+97%
`airbyte/mcp/int...nc_status_ui.py`	0%	97%	+97%

Python / code-coverage/pytest-unit

The overall line coverage in commit 1a7ea03 in the caleb/mcp-datadog-er... branch is 75%. Line coverage data for the release-candidate/v1 branch is not yet available.

Show a line coverage summary of the most covered files.

File	`release-candidate/v1`	`caleb/mcp-datadog-er...` 1a7ea03	+/-
`airbyte/cloud/models.py`	—	97%	—
`airbyte/mcp/_sso_auth.py`	—	96%	—
`airbyte/_direct...ctors/models.py`	—	94%	—
`airbyte/cloud/client.py`	—	90%	—
`airbyte/exceptions.py`	—	89%	—
`airbyte/mcp/cloud.py`	—	83%	—
`airbyte/progress.py`	—	80%	—
`airbyte/cloud/connectors.py`	—	75%	—
`airbyte/_util/api_util.py`	—	68%	—
`airbyte/shared/...ql_processor.py`	—	68%	—

_{Updated October 10, 2026 06:52 UTC}

…rary Merge #1354 (classify tool errors on native Datadog spans) into the branch that pins the fastmcp-extensions prerelease. The classification code is #1354's; the shared `ToolCallFacts` wiring from this branch is dropped in its favour. Adjusted for the pinned library: - `error.cause_types` is a tuple of class names, written as a JSON array under `airbyte.mcp.error.cause_types`. - `airbyte.mcp.upstream.status_code` is a span metric, so numeric queries work; the error message reads it from there. - An exception no rule recognises is `unclassified` / `unknown`, no longer `internal` / `server`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…g backends The slug lookup moves to _error_handling and is passed to the library as the error_reason hook, so the OTel backends export it too. The Datadog tag is renamed from error.problem_type to error.reason to match. Pins the fastmcp-extensions prerelease that adds the hook. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…ns like the OTel backend Only FastMCP's ToolError wrapper is unwrapped, so its ValidationError is classified as invalid_arguments instead of the chained pydantic error. A name FastMCP rejects reports outcome and category unknown_tool. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Copilot

🟡 Changes recommended

The production dependency remains pinned to an unreleased build, and reason extraction can export titles outside its documented generic-error case.

2 open findings

What changed in this PR

Adds bounded MCP tool-error classification to Datadog and OTel telemetry while preventing sensitive error details from being exported.

Changes:

Adds error categories, fault attribution, status codes, cause types, and reason slugs.
Classifies sync failures and wraps definition lookup SDK errors.
Expands telemetry and API error tests.

File	Description
`airbyte/mcp/_datadog.py`	Adds Datadog error classification and messages.
`airbyte/mcp/_otel.py`	Configures classifier and reason hooks.
`airbyte/mcp/_error_handling.py`	Implements sync classification and reason extraction.
`airbyte/_util/api_util.py`	Exposes response bodies and wraps definition errors.
`tests/unit_tests/test_mcp_datadog.py`	Tests Datadog classification and redaction.
`tests/unit_tests/test_mcp_otel.py`	Tests OTel hook configuration.
`tests/unit_tests/test_mcp_tool_errors.py`	Tests classifiers and ID filtering.
`tests/unit_tests/test_cloud_api_util.py`	Tests definition error wrapping.
`pyproject.toml`	Pins the development dependency.
`uv.lock`	Locks the development dependency artifacts.

🧠 Review effort: Balanced

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Aaron ("AJ") Steers (aaronsteers)

Approved. Looks good for my part.

Pending review and resolution of bot feedback (which I didn't review yet), I think this is good to merge+ship.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

This reverts commit 82d5db0.

…lues Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

…only for generic types Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

Caleb Evans (calebevans-ab) · 2026-10-10T05:59:42Z

Codex (@codex) review

chatgpt-codex-connector · 2026-10-10T05:59:58Z

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-10T06:33:47.083373Z	`1738378`	Manual request

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "Codex (@codex) review" or "Codex (@codex) security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 1ab4f5bf93

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "Codex (@codex) review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "Codex (@codex) address that feedback".

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

Caleb Evans (calebevans-ab) · 2026-10-10T06:27:11Z

Codex (@codex) review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 1738378511

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "Codex (@codex) review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "Codex (@codex) address that feedback".

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

Cam Kennedy (cjkenned) and others added 2 commits October 9, 2026 13:50

Caleb Evans (calebevans-ab) mentioned this pull request Oct 9, 2026

fix(telemetry): report failed tool calls honestly on the span and the event airbytehq/fastmcp-extensions#148

Merged

Caleb Evans (calebevans-ab) and others added 2 commits October 9, 2026 14:26

Caleb Evans (calebevans-ab) changed the title ~~fix(mcp): add failure category, fault, and cause to the Datadog tool span~~ feat(mcp): classify tool errors on Datadog spans Oct 9, 2026

Caleb Evans (calebevans-ab) and others added 2 commits October 9, 2026 14:39

fix(mcp): never export an ID-shaped problem slug as the error reason

f3a5770

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Aaron ("AJ") Steers (aaronsteers) marked this pull request as ready for review October 9, 2026 22:44

Copilot AI balanced review requested due to automatic review settings October 9, 2026 22:44

Copilot started reviewing on behalf of Aaron ("AJ") Steers (aaronsteers) October 9, 2026 22:45

View session

Copilot AI reviewed Oct 9, 2026

View reviewed changes

Comment thread pyproject.toml Outdated

Comment thread airbyte/mcp/_error_handling.py Outdated

Aaron ("AJ") Steers (aaronsteers) approved these changes Oct 9, 2026

View reviewed changes

Aaron ("AJ") Steers (aaronsteers) reviewed Oct 9, 2026

View reviewed changes

Comment thread pyproject.toml Outdated

Aaron ("AJ") Steers (aaronsteers) and others added 8 commits October 9, 2026 15:53

Apply suggestion from @aaronsteers

bfceadb

feat(mcp): export a frames-only error stack on Datadog tool spans

d74eeb3

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

chore(ci): suppress intentional release pin TODO

82d5db0

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

Revert "chore(ci): suppress intentional release pin TODO"

a0f60ed

This reverts commit 82d5db0.

feat(mcp): give agents actionable Cloud API errors and mask echoed va…

a7cd0d3

…lues Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

fix(mcp): match Cloud problem types in every published format

a16d535

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

fix(mcp): mask echoed values as whole tokens and read problem titles …

c6ae9d2

…only for generic types Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

refactor(mcp): move Cloud error messages into a YAML data file

bf6340b

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

devin-ai-integration Bot changed the title ~~feat(mcp): classify tool errors on Datadog spans~~ feat(mcp): classify tool errors on spans and give agents actionable, masked error text Oct 10, 2026

Caleb Evans (calebevans-ab) and others added 6 commits October 10, 2026 04:41

fix(mcp): show only known Cloud error types in agent messages

c5535d8

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

docs(mcp): document how to maintain Cloud error messages

bfc33b5

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

fix(mcp): align agent error handling and API boundary

b20f1c5

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

chore(deps): repin fastmcp-extensions prerelease

0831292

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

fix(mcp): tag raised tool-call errors

132b7db

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

chore(deps): pin fastmcp-extensions>=0.34.1

1ab4f5b

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

chatgpt-codex-connector Bot reviewed Oct 10, 2026

View reviewed changes

Caleb Evans (calebevans-ab) and others added 2 commits October 10, 2026 06:21

fix(mcp): close remaining agent-text and telemetry gaps from review

1fc2d5f

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

chore(deps): pin fastmcp-extensions>=0.34.2

1738378

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

chatgpt-codex-connector Bot reviewed Oct 10, 2026

View reviewed changes

Comment thread airbyte/mcp/_error_handling.py

Comment thread airbyte/_connector_base.py Outdated

Comment thread airbyte/mcp/_otel.py

Comment thread airbyte/mcp/_error_handling.py

fix(mcp): harden error telemetry and validation paths

1a7ea03

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

Caleb Evans (calebevans-ab) merged commit d67c5c5 into release-candidate/v1 Oct 10, 2026
29 checks passed

Caleb Evans (calebevans-ab) deleted the caleb/mcp-datadog-error-facts branch October 10, 2026 06:56

Conversation

Caleb Evans (calebevans-ab) commented Oct 9, 2026 • edited by devin-ai-integration Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What a failed span gains

Changes

Tracing

Agent-facing error text

Depends on

Verified locally

Heads-up

Uh oh!

github-code-quality Bot commented Oct 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code Coverage Overview

Python / code-coverage/pytest-no-creds

Python / code-coverage/pytest

Python / code-coverage/pytest-unit

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Aaron ("AJ") Steers (aaronsteers) left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Caleb Evans (calebevans-ab) commented Oct 10, 2026

Uh oh!

chatgpt-codex-connector Bot commented Oct 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codex Review Summary

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Caleb Evans (calebevans-ab) commented Oct 10, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Caleb Evans (calebevans-ab) commented Oct 9, 2026 •

edited by devin-ai-integration Bot

Loading

github-code-quality Bot commented Oct 9, 2026 •

edited

Loading

chatgpt-codex-connector Bot commented Oct 10, 2026 •

edited

Loading