Repository navigation
feat(mcp-instrumentation): suppress HTTP and initialization spans - #695
Merged
Merged
Conversation
liustve
force-pushed
the
mcp-enhancements
branch
2 times, most recently
from
March 30, 2026 05:21
1020f83 to
81dc242
Compare
liustve
changed the title
- Suppress redundant HTTP/ASGI spans on MCP client and server to reduce trace noise - Client: wrap streamable_http and sse transports with suppress_http_instrumentation() - Server: inject ASGI middleware into FastMCP.streamable_http_app() and FastMCP.sse_app() - Patch OpenTelemetryMiddleware.__call__ to respect suppress_http_instrumentation context - Skip creating MCP spans for initialize and notifications/initialized handshake messages - Controlled by OTEL_MCP_SUPPRESS_HTTP_INSTRUMENTATION env var, defaults to true
liustve
force-pushed
the
mcp-enhancements
branch
from
March 30, 2026 19:16
891387c to
9834328
Compare
wangzlei
previously approved these changes
Mar 30, 2026
wangzlei
approved these changes
Mar 30, 2026
jaredcnance
added a commit
to jaredcnance/aws-otel-python-instrumentation
that referenced
this pull request
Jul 19, 2026
… headers Even with OTEL_MCP_SUPPRESS_HTTP_INSTRUMENTATION=true (the default from request headers so MCP servers that read trace context only from HTTP headers can join the caller's trace. The default from aws-observability#695 correctly deduplicates the redundant HTTP CLIENT span, but it does so via `suppress_http_instrumentation()`, which also short-circuits the HTTP client instrumentation's own header injection — the only mechanism that would set `traceparent` on the wire. MCP servers that only participate in HTTP-header propagation (API Gateways, service meshes, non-Python MCP servers, any HTTP endpoint without ADOT MCP instrumentation) therefore receive no context and start a fresh trace, silently breaking E2E propagation for a large class of real-world topologies. Reported in aws-observability#826 with before/after trace evidence. This is compatible with the OTel MCP semconv (https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/mcp.md): > MCP server instrumentation SHOULD, by default, use context extracted > from MCP `params._meta` as a parent for MCP server span and SHOULD > link current ambient context, if it's present. The spec designates `_meta` as the MCP parent channel and HTTP context as an orthogonal transport signal (a span link on ADOT-instrumented servers). Nothing in the spec forbids injecting into HTTP headers; the existing ADOT server-side wrapper only extracts from `_meta`, so double-parenting is not a risk. This change adds HTTP-header propagation as a compatible fallback for non-ADOT servers while leaving the existing `_meta` propagation unchanged. Implementation: `ClientWrapper.wrap_http_client` now wraps the transport's `httpx_client_factory` (or supplies its own pre-built `http_client` for `streamable_http_client(url, http_client=...)`) so every httpx.AsyncClient the MCP transport uses has a request event hook that calls `self._propagators.inject(carrier=request.headers)`. The hook takes the OTel context at request-send time, so per-tool-call parents are honored correctly across a shared MCP session. If httpx or MCP is not importable, or the wrapped function signature doesn't match, the patcher no-ops silently. Tests: - `test_http_headers_carry_trace_context_regardless_of_suppression` exercises `OTEL_MCP_SUPPRESS_HTTP_INSTRUMENTATION` in {true, false, default} and asserts every outgoing MCP HTTP request carries a well-formed W3C `traceparent` header. - `test_http_headers_prebuilt_client_gets_hook` verifies the hook is installed on a user-supplied pre-built `http_client` (the newer `streamable_http_client(url, http_client=...)` API). Existing tests (span suppression, ping suppression, active-parent respect, tool/prompt/resource spans, error handling across stdio+http+sse transports) continue to pass unchanged. Fixes: aws-observability#826
liustve
added a commit
that referenced
this pull request
Jul 22, 2026
… headers (#827) ## Summary Even with `OTEL_MCP_SUPPRESS_HTTP_INSTRUMENTATION=true` (the default from #695), inject `traceparent`/`tracestate`/`baggage` into the outgoing HTTP request headers so MCP servers that read trace context only from HTTP headers can join the caller's trace. Fixes #826. ## Background The default from #695 correctly deduplicates the redundant HTTP CLIENT span, but it does so via `suppress_http_instrumentation()`, which also short-circuits the HTTP client instrumentation's own header injection — the only mechanism that would otherwise set `traceparent` on the wire. MCP servers that only participate in HTTP-header propagation (API Gateways, service meshes, non-Python MCP servers, any HTTP endpoint without ADOT MCP instrumentation) therefore receive no context and start a fresh trace, silently breaking E2E propagation for a large class of real-world topologies. #826 has before/after trace evidence from a real AWS deployment showing the client trace dead-ending at `mcp.session` while every server-side hop lands on its own fresh trace ID. ## Design Compatible with the [OTel MCP semantic convention](https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/mcp.md): > MCP server instrumentation SHOULD, by default, use context extracted from MCP `params._meta` as a parent for MCP server span and SHOULD link current ambient context, if it's present. The spec designates `_meta` as the MCP parent channel and HTTP context as an orthogonal transport signal (a span link on ADOT-instrumented servers). Nothing in the spec forbids injecting into HTTP headers as well; the existing ADOT server-side wrapper only extracts from `_meta`, so double-parenting on ADOT-instrumented servers is not a risk. This change adds HTTP-header propagation as a compatible fallback for non-ADOT servers while leaving the existing `_meta` propagation and the span-suppression semantics unchanged. ## Implementation `ClientWrapper.wrap_http_client` now wraps the transport's `httpx_client_factory` (or supplies its own pre-built `http_client` for the newer `streamable_http_client(url, http_client=...)` API) so every `httpx.AsyncClient` the MCP transport uses has a request event hook that calls `self._propagators.inject(carrier=request.headers)`. The hook takes the OTel context *at request-send time*, so per-tool-call parents (which change during a shared MCP session) are honored correctly. If httpx or MCP is not importable, or the wrapped function signature doesn't match, the patcher no-ops silently. The change is purely additive with respect to `OTEL_MCP_SUPPRESS_HTTP_INSTRUMENTATION`: | `OTEL_MCP_SUPPRESS_HTTP_INSTRUMENTATION` | HTTP CLIENT span | `traceparent` HTTP header | | --- | --- | --- | | `true` (default) — before this PR | suppressed ✓ | **absent ✗** (bug) | | `true` (default) — after this PR | suppressed ✓ | **present ✓** | | `false` — before this PR | present | present | | `false` — after this PR | present | present | Span suppression, ping suppression, active-parent respect, tool/prompt/resource spans, and error handling across stdio+http+sse transports are unchanged. ## Testing **New tests** in `test_mcp_instrumentor.py`: - `test_http_headers_carry_trace_context_regardless_of_suppression` — exercises `OTEL_MCP_SUPPRESS_HTTP_INSTRUMENTATION` in `{true, false, default}` and asserts every outgoing MCP HTTP request carries a well-formed W3C `traceparent` header (validates format: `00-<32hex trace-id>-<16hex span-id>-<2hex flags>`). - `test_http_headers_prebuilt_client_gets_hook` — verifies the hook is installed on a user-supplied pre-built `http_client` (newer `streamable_http_client(url, http_client=...)` API). **Existing tests** (15 total including the two new ones): ``` $ python -m pytest test_mcp_instrumentor.py ======================== 15 passed, 1 warning in 20.89s ======================== ``` **Lint**: ``` $ black --check aws-opentelemetry-distro/src/.../_wrappers.py .../test_mcp_instrumentor.py All done! ✨ 🍰 ✨ 2 files would be left unchanged. $ flake8 aws-opentelemetry-distro/src/.../_wrappers.py .../test_mcp_instrumentor.py (no output) $ pylint --rcfile pyproject.toml aws-opentelemetry-distro/src/.../_wrappers.py Your code has been rated at 9.01/10 (previous run: 9.01/10, +0.00) ``` ## Related PRs - #683 — first added HTTP context propagation for MCP requests (via `_meta` + `_handle_post_request` restoration) - #695 — introduced `OTEL_MCP_SUPPRESS_HTTP_INSTRUMENTATION` defaulting to `true` (this is the PR whose default silently broke non-ADOT-server topologies) --------- Co-authored-by: Steve Liu <liustve@amazon.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description of changes:
Suppress redundant HTTP/ASGI spans on MCP client and server by default
Patch
OpenTelemetryMiddleware.__call__to respectsuppress_http_instrumentationcontextnote: SHOULD remove this patch once feat(asgi): respect suppress_http_instrumentation in ASGI middleware open-telemetry/opentelemetry-python-contrib#4375 is merged
Skip creating MCP spans for initialize and notifications/initialized handshake messages
Introduce
OTEL_MCP_SUPPRESS_HTTP_INSTRUMENTATIONenv var, defaults to trueBy submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.