feat: Add support for upstream trace headers as request ID

[captainp1an3t]

Correlate Athens request IDs with distributed traces

Problem

When Athens runs behind a service mesh (Istio, Linkerd) or API gateway, there's no way to correlate Athens logs with distributed traces. The mesh injects trace headers (traceparent, x-b3-traceid) into incoming requests, but Athens ignores them and generates its own request ID. This forces teams to manually correlate logs and traces by timestamp and path.

Solution

Use the OpenTelemetry propagator to extract trace context from incoming requests and adopt the trace ID as the Athens request ID. If trace headers are present, the trace ID is used; otherwise the existing behavior is preserved.

Note: The original PR was proposed prior to #2138 getting merged. The conversation/comments history may appear disjointed.

Capture Order of the incoming trace id:

1. If traceparent (W3C) or b3/X-B3-TraceId (Zipkin) headers → use it
2. Else If Athens-Request-ID header (legacy) → use it
3. Else No relevant headers → generate UUID

What changed

File	Change
pkg/middleware/requestid.go	Extract remote span context via OTel propagator; use trace ID if present, else fall back to Athens-Request-ID/UUID
pkg/middleware/requestid_test.go	Tests covering all priority paths
pkg/observ/observ.go	Extract propagator registration into RegisterPropagator() (W3C TraceContext + B3); called unconditionally so trace correlation works even without a trace exporter
cmd/proxy/actions/app.go	Call observ.RegisterPropagator() unconditionally before exporter setup
go.mod / go.sum	Add go.opentelemetry.io/contrib/propagators/b3

Design note

There was an attempt to satisfy this comment HOWEVER otelhttp.NewHandler (which wraps the router) always creates a span with a valid trace ID, even when no trace headers are present. Thus there is no way to distinguish "trace ID came from an upstream header" vs. "otelhttp auto-generated one" simply by looking span alone.

So the middleware does not read the trace ID from the active span context (trace.SpanFromContext). Instead, the middleware re-invokes the global OTel propagator against the request headers to detect whether a remote trace context was actually propagated. This is format-agnostic (supports any propagation format registered in the composite propagator) and avoids manual header parsing, but it is technically re-extracting from headers rather than purely deriving from the span.

Backwards compatibility

No breaking changes: Existing deployments that use Athens-Request-ID or no trace headers see identical behavior. Only when trace headers are present does behavior change (request ID becomes the trace ID instead of a UUID).

How I Tested It

Unit tests

Added a few unit tests for failure cases. Ran full suite.

go clean -testcache && go test ./...
?       github.com/gomods/athens/cmd/proxy      [no test files]
ok      github.com/gomods/athens/cmd/proxy/actions      1.728s
ok      github.com/gomods/athens/internal/shutdown      0.455s
?       github.com/gomods/athens/pkg/build      [no test files]
ok      github.com/gomods/athens/pkg/config     2.463s
ok      github.com/gomods/athens/pkg/download   19.722s
ok      github.com/gomods/athens/pkg/download/addons    1.955s
ok      github.com/gomods/athens/pkg/download/mode      1.133s
ok      github.com/gomods/athens/pkg/errors     0.984s
?       github.com/gomods/athens/pkg/index      [no test files]
?       github.com/gomods/athens/pkg/index/compliance   [no test files]
ok      github.com/gomods/athens/pkg/index/mem  2.172s
ok      github.com/gomods/athens/pkg/index/mysql        2.266s
?       github.com/gomods/athens/pkg/index/nop  [no test files]
ok      github.com/gomods/athens/pkg/index/postgres     0.814s
ok      github.com/gomods/athens/pkg/log        0.626s
ok      github.com/gomods/athens/pkg/middleware 2.854s
ok      github.com/gomods/athens/pkg/module     38.679s
ok      github.com/gomods/athens/pkg/observ     2.860s
ok      github.com/gomods/athens/pkg/paths      2.796s
?       github.com/gomods/athens/pkg/requestid  [no test files]
ok      github.com/gomods/athens/pkg/stash      39.901s
?       github.com/gomods/athens/pkg/storage    [no test files]
ok      github.com/gomods/athens/pkg/storage/azureblob  3.138s
?       github.com/gomods/athens/pkg/storage/compliance [no test files]
ok      github.com/gomods/athens/pkg/storage/external   2.934s
ok      github.com/gomods/athens/pkg/storage/fs 2.845s
ok      github.com/gomods/athens/pkg/storage/gcp        3.008s
?       github.com/gomods/athens/pkg/storage/mem        [no test files]
ok      github.com/gomods/athens/pkg/storage/minio      3.074s
ok      github.com/gomods/athens/pkg/storage/module     5.071s
ok      github.com/gomods/athens/pkg/storage/mongo      2.767s
ok      github.com/gomods/athens/pkg/storage/s3 2.543s

Integration Tests

Ran Athens in Docker (make run-docker) with Mongo 6 and Jaeger.

Note: OTLP tracing was enabled in the test environment but is not required as trace correlation works regardless of exporter configuration.

Test 1: W3C traceparent

$ curl -s -H "traceparent: 00-4bf92f3577b6a814af67ab2d6fc0f4e1-00f067aa0ba902b7-01" localhost:3000 > /dev/null
INFO[...]: incoming request  http-method=GET http-path=/ http-status=200 request-id=4bf92f3577b6a814af67ab2d6fc0f4e1

Test 2: B3 multi-header

#$ curl -s -H "X-B3-TraceId: 463ac35c9f6413ad48485a3953bb6124" \
       -H "X-B3-SpanId: 0020000000000001" -H "X-B3-Sampled: 1" localhost:3000 > /dev/null
INFO[...]: incoming request  http-method=GET http-path=/ http-status=200 request-id=463ac35c9f6413ad48485a3953bb6124

Test 3: B3 single header

$ curl -s -H "b3: 80f198ee56343ba864fe8b2a57d3eff7-e457b5a2e4d86bd1-1" localhost:3000 > /dev/null
INFO[...]: incoming request  http-method=GET http-path=/ http-status=200 request-id=80f198ee56343ba864fe8b2a57d3eff7

Test 4: Both W3C and B3 present → W3C wins

$ curl -s -H "traceparent: 00-aaaa1111bbbb2222cccc3333dddd4444-00f067aa0ba902b7-01" \
       -H "X-B3-TraceId: 1111222233334444aaaabbbbccccdddd" \
       -H "X-B3-SpanId: 0020000000000001" -H "X-B3-Sampled: 1" localhost:3000 > /dev/null
INFO[...]: incoming request  http-method=GET http-path=/ http-status=200 request-id=aaaa1111bbbb2222cccc3333dddd4444

Test 5: Athens-Request-ID header (no trace headers)

$ curl -s -H "Athens-Request-ID: my-custom-id-123" localhost:3000 > /dev/null
INFO[...]: incoming request  http-method=GET http-path=/ http-status=200 request-id=my-custom-id-123

Test 6: No headers → UUID fallback

$ curl -s localhost:3000 > /dev/null
INFO[...]: incoming request  http-method=GET http-path=/ http-status=200 request-id=d39cf1e4-bf46-4a9a-8ece-9d026b5f2c35

[matt0x6F]

Thanks for your contribution to Athens!

This is good work, but I'm curious if we could rather derive the request ID from the active span context rather than switching on header values as such.

[matt0x6F]

B3 supports multiple and single header propagation: https://github.com/openzipkin/b3-propagation#single-header

[captainp1an3t]

Thanks for your contribution to Athens!

This is good work, but I'm curious if we could rather derive the request ID from the active span context rather than switching on header values as such.

I'll take a look if this is possible. My assumption is that it depends on what the capabilities of the opencensus middleware are, but I'll see what I can do.

[captainp1an3t]

Thanks for your contribution to Athens!
This is good work, but I'm curious if we could rather derive the request ID from the active span context rather than switching on header values as such.

I'll take a look if this is possible. My assumption is that it depends on what the capabilities of the opencensus middleware are, but I'll see what I can do.

I haven't abandoned this, but I need to investigate how #2138 impacts my changes.

[captainp1an3t]

@matt0x6F Can you take a fresh look at this?