Skip to content

feat(mcp): send each tool call line to Axiom - #94

Merged
Gintm merged 2 commits into
developfrom
feat/mcp-tool-calls-to-axiom
Nov 12, 2026
Merged

Gintm merged 2 commits into
developfrom
feat/mcp-tool-calls-to-axiom

Conversation

@Gintm

@Gintm Gintm commented Nov 12, 2026

Copy link
Copy Markdown
Contributor

Why

Since Nov 12 00:00 UTC the Sentry organization is over its span quota (span_usage_exceeded), so the MCP's SDK backs off and drops every span (backpressure, ~1M per 6 h). We cannot read tool-call latency, result size or the new mcp.auth tag, and the deploys of the MCP speed work (py#88 to py#93, API#8251) cannot be measured. The same data is already printed per call as an mcp_tool_call JSON line on Railway's stdout, which no tool can query.

What

  • AxiomIngestHandler on the late.mcp.tools logger: buffers each mcp_tool_call line and posts batches to https://api.axiom.co/v1/datasets/<AXIOM_DATASET>/ingest?timestamp-field=ts every 5 s (or at 1,000 lines) from a background thread.
  • Adds release (commit), environment and worker (pid) to each event, so restarts and per-worker behaviour are visible.
  • Never slows or fails a tool call: a failed or rejected batch is logged and dropped; over 20,000 waiting lines, new ones are dropped and counted in an mcp_axiom_dropped event; buffered lines are sent on shutdown.
  • Off unless both AXIOM_TOKEN and AXIOM_DATASET are set. Documented in docs/HTTP_DEPLOYMENT.md.

Follow-ups

  • Create the dataset (zernio-mcp) and an ingest-only token in Axiom, and set AXIOM_TOKEN and AXIOM_DATASET on the Railway mcp-server service.
  • The Sentry span quota itself (budget, or cutting schedule-posts-api ingest) is a separate decision.

🤖 Generated with Claude Code

Gintm and others added 2 commits November 12, 2026 17:10
…OM_DATASET are set

Sentry stopped accepting spans when the organization's quota ran out, so
tool-call latency, size and auth outcomes could not be measured. The same
mcp_tool_call line printed to stdout now also goes to an Axiom dataset,
batched every 5 s from a background thread, dropped (and counted) instead
of ever slowing a tool call.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…he log pump

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Gintm

Gintm commented Nov 12, 2026

Copy link
Copy Markdown
Contributor Author

Review (code-review high --fix)

Finding Result
An error other than httpx.HTTPError (e.g. InvalidURL from a bad dataset name) killed the ingest thread for good Fixed: any send error is logged and the next flush still sends (test)
The log pump wrapped the Axiom handler too, two pump items per tool call against a cap of 100/s per process Fixed: handlers marked emits_without_blocking are left unwrapped (test)
JSON serialized by telemetry and parsed back by the handler Skipped: one json.loads per call, keeps telemetry unaware of sinks
close() at exit can wait up to 10 s on a slow Axiom Skipped: at most ~5 s of buffered lines, normally one batch

QA

Check Result
Real POST to https://api.axiom.co/v1/datasets/<ds>/ingest?timestamp-field=ts with a fake token Axiom answers 401; handler logs rejected 1 events: HTTP 401, raises nothing; emit 0.01 ms, flush 319 ms off the request path
End-to-end: a tool call through ToolTelemetryMiddleware lands as one mcp_tool_call event Pass
Full suite 1,799 passed
Real ingest into a dataset Pending: needs the dataset, an ingest token, and AXIOM_TOKEN + AXIOM_DATASET on Railway

Metrics

Metric Before Expected after
MCP tool calls queryable 0 since Nov 12 00:00 UTC (Sentry over quota, ~1M spans dropped per 6 h by backpressure) every call, about 700K events/day (~300 bytes each, ~0.2 GB/day ingest)
Tool latency, raw_chars, auth, outcome, status_code per tool Not readable Readable in APL, per release and worker

Post-deploy check: hourly count of mcp_tool_call events in the dataset against the hourly count of late-python-sdk calls from SJC on the Worker (control); they should track each other. Then p50/p95 duration_ms by tool and the auth split, for A1 and B1.

@Gintm
Gintm merged commit 478990c into develop Nov 12, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant