Skip to content

refactor(mcp): classify traced tool args from a rules file - #1412

Draft
Aaron ("AJ") Steers (aaronsteers) wants to merge 5 commits into
release-candidate/v1from
devin/1792462628-arg-trace-rules
Draft

Aaron ("AJ") Steers (aaronsteers) wants to merge 5 commits into
release-candidate/v1from
devin/1792462628-arg-trace-rules

Conversation

@aaronsteers

Copy link
Copy Markdown
Member

Summary

MCP tool-argument trace classes now come from one rules file, airbyte/mcp/arg_trace_rules.yaml, instead of TraceArg markers on each tool signature plus a hand-regenerated JSON snapshot. Requested by Aaron ("AJ") Steers (@aaronsteers) in #1392 (comment) (option 2). Depends on airbytehq/fastmcp-extensions#151.

  1. Managed artifact: arg_trace_rules.yaml holds ordered name globs (*_id: hash, *_name: fingerprint, workspace_id: omit), plus tool:arg keys for real exceptions (get_stream_previews:streams). It's loaded by _arg_trace_rules.py and passed to both tracers: ToolTracingConfig(arg_rules=...) in _otel.py and ArgTracer(rules=...) in _datadog.py.
  2. No markers: all 135 TraceArg markers are removed from airbyte/mcp/**.
  3. Generated artifact: tests/unit_tests/fixtures/arg_trace_plan.json is deleted. poe mcp-arg-trace-plan writes the full plan, with what decided each arg, to the gitignored arg_trace_plan.json.
  4. CI failure tells you what to edit: test_every_arg_is_classified_by_a_rule fails when an arg falls to the type default or carries a marker, and says to add a rule to the YAML. test_no_dead_rules fails when a rule matches nothing. test_datadog_and_otel_classification_agree now covers each tool's full declared params, including the intent skip.
  5. AGENTS.md has a new ## MCP Argument Tracing section.

bool/Literal/Enum args are still recorded as values, and dict/model args as presence, without a rule. A rule overrides the secret-name heuristic: defer_credentials: value.

Evidence

Check Result
Generated plan vs RC's deleted fixture (trusted execution, 61 tools) identical
test_mcp_arg_trace_parity.py, test_mcp_datadog.py, test_mcp_otel.py 88 passed
Datadog-vs-OTel test without the intent skip fails (proves the guard)
Wheel contents includes airbyte/mcp/arg_trace_rules.yaml
ruff check / format, pyrefly pass

Before merge

Link to Devin session: https://app.devin.ai/sessions/4d1ece6885c44c52a07034e025db11e6
Open in Devin Desktop: https://app.devin.ai/desktop/session/4d1ece6885c44c52a07034e025db11e6?variant=devin

devin-ai-integration Bot and others added 2 commits October 20, 2026 02:25
Co-Authored-By: AJ Steers <aj@airbyte.io>
Co-Authored-By: AJ Steers <aj@airbyte.io>
@devin-ai-integration

Copy link
Copy Markdown
Contributor

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

  • Disable automatic comment, CI, and merge conflict monitoring

@github-actions

Copy link
Copy Markdown

👋 Greetings, Airbyte Team Member!

Here are some helpful tips and reminders for your convenience.

💡 Show Tips and Tricks

Testing This PyAirbyte Version

You can test this version of PyAirbyte using the following:

# Run PyAirbyte CLI from this branch:
uvx --from 'git+https://github.com/airbytehq/PyAirbyte.git@devin/1792462628-arg-trace-rules' pyairbyte --help

# Install PyAirbyte from this branch for development:
pip install 'git+https://github.com/airbytehq/PyAirbyte.git@devin/1792462628-arg-trace-rules'

PR Slash Commands

Airbyte Maintainers can execute the following slash commands on your PR:

  • /fix-pr - Fixes most formatting and linting issues
  • /uv-lock - Updates uv.lock file
  • /test-pr - Runs tests with the updated PyAirbyte
  • /prerelease - Builds and publishes a prerelease version to PyPI
  • /evals - Runs the Cloud MCP evals that use the MCP tools this PR changed (/evals full=true runs the full suite)
📚 Show Repo Guidance

Helpful Resources

Community Support

Questions? Join the #pyairbyte channel in our Slack workspace.

📝 Edit this welcome message.

@github-code-quality

github-code-quality Bot commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

Code coverage overview

Languages: Python

Python / code-coverage/pytest-no-creds

release-candidate/v1 d9f652f devin/1792462628-arg... 9414ed2 +/-
65% 79% +14%
View coverage of the 10 most impacted files.
File release-candidate/v1 d9f652f devin/1792462628-arg... 9414ed2 +/-
airbyte/_util/api_util.py 36% 67% +31%
airbyte/mcp/cloud.py 52% 85% +33%
airbyte/cloud/connectors.py 34% 79% +45%
airbyte/mcp/_datadog.py 0% 50% +50%
airbyte/mcp/_user_identity.py 0% 93% +93%
airbyte/mcp/_telemetry.py 0% 94% +94%
airbyte/_direct...ctors/models.py 0% 95% +95%
airbyte/mcp/_sso_auth.py 0% 96% +96%
airbyte/cloud/models.py 0% 97% +97%
airbyte/mcp/int...nc_status_ui.py 0% 97% +97%

Python / code-coverage/pytest

release-candidate/v1 d9f652f devin/1792462628-arg... 9414ed2 +/-
71% 82% +11%
View coverage of the 10 most impacted files.
File release-candidate/v1 d9f652f devin/1792462628-arg... 9414ed2 +/-
airbyte/cloud/connectors.py 51% 83% +32%
airbyte/mcp/cloud.py 52% 85% +33%
airbyte/mcp/_datadog.py 0% 50% +50%
airbyte/mcp/int..._registry_ui.py 0% 92% +92%
airbyte/mcp/_user_identity.py 0% 93% +93%
airbyte/mcp/_telemetry.py 0% 94% +94%
airbyte/_direct...ctors/models.py 0% 95% +95%
airbyte/mcp/_sso_auth.py 0% 96% +96%
airbyte/cloud/models.py 0% 97% +97%
airbyte/mcp/int...nc_status_ui.py 0% 97% +97%

Python / code-coverage/pytest-unit

release-candidate/v1 devin/1792462628-arg... 9414ed2 +/-
— 76% —

Line coverage data for the release-candidate/v1 branch is not yet available.

View coverage of the 10 most covered files.
File release-candidate/v1 devin/1792462628-arg... 9414ed2 +/-
airbyte/cloud/models.py — 97% —
airbyte/mcp/_sso_auth.py — 96% —
airbyte/_direct...ctors/models.py — 95% —
airbyte/cloud/client.py — 90% —
airbyte/exceptions.py — 90% —
airbyte/mcp/cloud.py — 85% —
airbyte/progress.py — 80% —
airbyte/cloud/connectors.py — 79% —
airbyte/shared/...ql_processor.py — 68% —
airbyte/_util/api_util.py — 67% —

ℹ️ Calculations represent line coverage. Results may change as calculations are updated.


Updated October 20, 2026 03:18 UTC

devin-ai-integration Bot and others added 2 commits October 20, 2026 02:42
Co-Authored-By: AJ Steers <aj@airbyte.io>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The direct Git dependency blocks PyPI publishing and needs replacement with a released version range.

1 open issue
What changed in this PR

Centralizes MCP argument-tracing classifications in YAML shared by the OpenTelemetry and Datadog backends.

Changes:

  • Replaces tool-signature markers with ordered tracing rules.
  • Replaces the committed trace snapshot with plan generation and classification checks.
  • Updates packaging and contributor guidance.
File Description
uv.lock Locks the rules-capable Git dependency.
tests/​unit_tests/​test_mcp_arg_trace_parity.py Checks rule coverage and backend parity.
tests/​unit_tests/​fixtures/​arg_trace_plan.json Removes the committed snapshot.
pyproject.toml Updates dependency, YAML packaging, and generation task.
airbyte/​mcp/​registry.py Removes tracing markers.
airbyte/​mcp/​local.py Removes tracing markers.
airbyte/​mcp/​interactive/​_workspace_sync_status_ui.py Removes workspace tracing marker.
airbyte/​mcp/​interactive/​_sync_history_ui.py Removes workspace tracing marker.
airbyte/​mcp/​interactive/​_registry_ui.py Removes tracing markers.
airbyte/​mcp/​guidance.py Removes tracing markers.
airbyte/​mcp/​cloud.py Removes tracing markers.
airbyte/​mcp/​arg_trace_rules.yaml Defines shared classification rules.
airbyte/​mcp/​_trace_plan.py Generates the plan with classification sources.
airbyte/​mcp/​_otel.py Supplies rules to OpenTelemetry tracing.
airbyte/​mcp/​_datadog.py Supplies rules to Datadog tracing.
airbyte/​mcp/​_arg_trace_rules.py Loads the YAML rules.
AGENTS.md Documents rule maintenance.
.gitignore Ignores the generated plan.

Review effort: Balanced

Comment thread pyproject.toml
"fastmcp>=4.0.9,<5.0",
"fastmcp-extensions>=0.35.0,<1.0.0",
# TK-TODO: Restore fastmcp-extensions>=<next release>,<1.0.0 after the rules release.
"fastmcp-extensions @ git+https://github.com/airbytehq/fastmcp-extensions.git@devin/1792462457-arg-trace-rules",

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. This is intentional while airbytehq/fastmcp-extensions#151 is unreleased, and the # TK-TODO marker makes the "Check for TK-TODO markers" job fail, which blocks merging until it's swapped. Once #151 ships, this line goes back to fastmcp-extensions>=<that release>,<1.0.0 and uv lock gets re-run. I'm leaving this thread open until then.

Written by Devin

…78d1f6961ac33335777557b5

Co-Authored-By: AJ Steers <aj@airbyte.io>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants