[jaredcnance]

Summary

Even with OTEL_MCP_SUPPRESS_HTTP_INSTRUMENTATION=true (the default from #695), inject traceparent/tracestate/baggage into the outgoing HTTP request headers so MCP servers that read trace context only from HTTP headers can join the caller's trace.

Fixes #826.

Background

The default from #695 correctly deduplicates the redundant HTTP CLIENT span, but it does so via suppress_http_instrumentation(), which also short-circuits the HTTP client instrumentation's own header injection — the only mechanism that would otherwise set traceparent on the wire. MCP servers that only participate in HTTP-header propagation (API Gateways, service meshes, non-Python MCP servers, any HTTP endpoint without ADOT MCP instrumentation) therefore receive no context and start a fresh trace, silently breaking E2E propagation for a large class of real-world topologies.

#826 has before/after trace evidence from a real AWS deployment showing the client trace dead-ending at mcp.session while every server-side hop lands on its own fresh trace ID.

Design

Compatible with the OTel MCP semantic convention:

MCP server instrumentation SHOULD, by default, use context extracted from MCP params._meta as a parent for MCP server span and SHOULD link current ambient context, if it's present.

The spec designates _meta as the MCP parent channel and HTTP context as an orthogonal transport signal (a span link on ADOT-instrumented servers). Nothing in the spec forbids injecting into HTTP headers as well; the existing ADOT server-side wrapper only extracts from _meta, so double-parenting on ADOT-instrumented servers is not a risk. This change adds HTTP-header propagation as a compatible fallback for non-ADOT servers while leaving the existing _meta propagation and the span-suppression semantics unchanged.

Implementation

ClientWrapper.wrap_http_client now wraps the transport's httpx_client_factory (or supplies its own pre-built http_client for the newer streamable_http_client(url, http_client=...) API) so every httpx.AsyncClient the MCP transport uses has a request event hook that calls self._propagators.inject(carrier=request.headers).

The hook takes the OTel context at request-send time, so per-tool-call parents (which change during a shared MCP session) are honored correctly. If httpx or MCP is not importable, or the wrapped function signature doesn't match, the patcher no-ops silently.

The change is purely additive with respect to OTEL_MCP_SUPPRESS_HTTP_INSTRUMENTATION:

OTEL_MCP_SUPPRESS_HTTP_INSTRUMENTATION	HTTP CLIENT span	traceparent HTTP header
true (default) — before this PR	suppressed ✓	absent ✗ (bug)
true (default) — after this PR	suppressed ✓	present ✓
false — before this PR	present	present
false — after this PR	present	present

Span suppression, ping suppression, active-parent respect, tool/prompt/resource spans, and error handling across stdio+http+sse transports are unchanged.

Testing

New tests in test_mcp_instrumentor.py:

• test_http_headers_carry_trace_context_regardless_of_suppression — exercises OTEL_MCP_SUPPRESS_HTTP_INSTRUMENTATION in {true, false, default} and asserts every outgoing MCP HTTP request carries a well-formed W3C traceparent header (validates format: 00-<32hex trace-id>-<16hex span-id>-<2hex flags>).
• test_http_headers_prebuilt_client_gets_hook — verifies the hook is installed on a user-supplied pre-built http_client (newer streamable_http_client(url, http_client=...) API).

Existing tests (15 total including the two new ones):

$ python -m pytest test_mcp_instrumentor.py
======================== 15 passed, 1 warning in 20.89s ========================

Lint:

$ black --check aws-opentelemetry-distro/src/.../_wrappers.py .../test_mcp_instrumentor.py
All done! ✨ 🍰 ✨
2 files would be left unchanged.

$ flake8 aws-opentelemetry-distro/src/.../_wrappers.py .../test_mcp_instrumentor.py
(no output)

$ pylint --rcfile pyproject.toml aws-opentelemetry-distro/src/.../_wrappers.py
Your code has been rated at 9.01/10 (previous run: 9.01/10, +0.00)

Related PRs

• feat: propagate HTTP context for MCP requests and prefix all span names with mcp #683 — first added HTTP context propagation for MCP requests (via _meta + _handle_post_request restoration)
• feat(mcp-instrumentation): suppress HTTP and initialization spans #695 — introduced OTEL_MCP_SUPPRESS_HTTP_INSTRUMENTATION defaulting to true (this is the PR whose default silently broke non-ADOT-server topologies)

[jaredcnance]

Let’s hold on merging this until next week as I want to run a few more e2e tests. Will post test results to the PR.

[jaredcnance]

Verified end-to-end on an AWS deployment where the MCP server is a Lambda Java service fronted by API Gateway with X-Ray active tracing. Before this change: client trace dead-ended at mcp.session, and each MCP tool call produced an orphan trace on the server side (fresh root at POST /mcp). After this change:

  [SERVER] AgentCore.Runtime.Invoke                       ← agent runtime root
  └─ [INTERNAL] execute_event_loop_cycle                  (agent loop)
     ├─ [INTERNAL] chat
     │  └─ [CLIENT] chat anthropic.claude-sonnet-4        (Bedrock)
     └─ [INTERNAL] execute_tool query_metrics             (MCP tool)
        └─ [CLIENT] mcp tools/call query_metrics             (httpx → traceparent header)
           └─ [SERVER] MetricsApi/prod                    (AWS::ApiGateway::Stage)
              └─ [CLIENT] Lambda                          (API GW integration)
                 └─ [SERVER] MetricsService-Frontend/LambdaService
                    └─ [SERVER] MetricsService-Frontend/LambdaExecutionEnvironment

Sample client span:

  {
    "name": "mcp tools/call query_metrics",
    "kind": "CLIENT",
    "traceId": "5f369767e659917c65a1a4573210866a",
    "spanId": "bb63e5278111c0c5",
    "parentSpanId": "74d96fd91d70c4e8",
    "durationMs": 15.3,
    "status": { "code": "OK" },
    "attributes": {
      "mcp.method.name": "tools/call",
      "mcp.protocol.version": "2025-11-25",
      "mcp.session.id": "e9a00860bd084ee88d80e6e990f6e3fd",
      "jsonrpc.request.id": "1",
      "gen_ai.tool.name": "query_metrics",
      "gen_ai.operation.name": "execute_tool",
      "gen_ai.tool.call.arguments": "{\"namespace\": \"AWS/Lambda\", \"metric_name\": \"Errors\"}",
      "gen_ai.tool.call.result": "…{\"datapoints\": [{\"timestamp\": \"2026-07-19T00:00:00Z\", \"value\": 42.0}]}… isError=False",
      "network.transport": "tcp",
      "server.address": "127.0.0.1",
      "server.port": 54499
    },
    "instrumentationScope": {
      "name": "amazon.opentelemetry.distro.instrumentation.mcp",
      "version": "0.18.0.dev0"
    }
  }

[liustve]

Thanks @jaredcnance for the changes and testing!
One suggestion: I think we should follow OTel botocore instrumentation pattern and wrap the header-prep function.

The MCP function to wrap would be is StreamableHTTPTransport._prepare_headers, which already runs under our attached _meta context.

under wrapper.py

def wrap_prepare_headers(self, wrapped, instance, args, kwargs):
    headers = wrapped(*args, **kwargs)
    if self._should_suppress_http_spans:  # else httpx injects it itself
        self._propagators.inject(carrier=headers)
    return headers

under _instrument() and _uninstrument() in:

try_wrap(
    _HTTP_MODULE,
    "StreamableHTTPTransport._prepare_headers",
    self._client_wrapper.wrap_prepare_headers,
)

if hasattr(streamable_http, "StreamableHTTPTransport"):
    try_unwrap(streamable_http.StreamableHTTPTransport, "_prepare_headers")

[jaredcnance]

sweet -- thanks for the pointers. PR is updated, let me know if this makes sense. verified end-to-end in deployed environment.