Smoke Test: OTEL Backends - 25918800907 #32366

Smoke Test: OTEL Backends

Overall Result: FAIL

Run: https://github.com/github/gh-aw/actions/runs/25918800907

Executive Summary

Local OTEL emission is partially working — a span for run 25918800907 was written to the local JSONL mirror, but 2 OTLP export errors were recorded, indicating the push to remote backends failed. Grafana Tempo shows zero spans across all service names in a 2-hour query window, consistent with push failures. Sentry project gh-aw exists and MCP access works, but no events-query tool is available to confirm span visibility.

Checklist

• Local send verification — span for run 25918800907 written to /tmp/gh-aw/otel.jsonl
• Local send verification — OTLP export errors = 2 (push failures recorded)
• Sentry query verification — project found but span visibility unconfirmable
• Grafana query verification — Tempo empty (0 service names in 2-hour window)

Evidence

Step 1: Local OTEL Emission — send_status = inconclusive

The local JSONL mirror contains a span tagged with the current run ID. However, /tmp/gh-aw/otlp-export-errors.count = 2, meaning two OTLP push batches failed. COPILOT_OTEL_FILE_EXPORTER_PATH is unset (file exporter not configured).

Step 2: Sentry — sentry_status = inconclusive

The Sentry MCP exposes find_organizations, find_projects, whoami, etc., but no trace/span search tool. The project exists. Whether spans arrived cannot be confirmed or denied — this is an observability gap in the query tooling.

Step 3: Grafana — grafana_status = fail

Query used: {resource.service.name="gh-aw"} via tempo_traceql-search. Both 30-minute and 2-hour windows returned {"traces":[]}. tempo_get-attribute-values for resource.service.name returned {}, confirming no spans of any service are in the Tempo backend for the last 2 hours. This is consistent with the 2 OTLP push failures.

Blockers

❌ OTLP push failures (emit-side)

/tmp/gh-aw/otlp-export-errors.count = 2. The spans were written locally but failed to reach the configured OTLP endpoint (OTEL_EXPORTER_OTLP_ENDPOINT). Check:

• The endpoint is reachable from the runner (firewall/network)
• The headers in OTEL_EXPORTER_OTLP_HEADERS are valid and unexpired
• GH_AW_OTLP_ENDPOINTS routing logic is correct

❌ Grafana Tempo completely empty (read-side)

No span data at all in the grafanacloud-traces Tempo datasource. This is the downstream effect of the OTLP push failures. Once push errors are resolved, Tempo should start receiving data.

⚠️ Sentry span-query tooling gap

The Sentry MCP server has no search_events, search_traces, or equivalent span-search tool. Sentry verification is blocked. A search_issues or get_sentry_resource tool for span/event queries would enable proper verification.

Run URL: https://github.com/github/gh-aw/actions/runs/25918800907

Generated by 🧪 Smoke OTEL Backends for issue #32351 · ● 4.9M · ◷

• expires on May 15, 2026, 3:00 PM UTC