[SC-17274] Emit Atryum activity as OpenTelemetry traces

[juanmleng]

Pull Request Description

What and why?

Atryum now reports its activity — every agent tool-call it governs and every AI-judge decision — as OpenTelemetry (OTEL) traces. OTEL is the industry-standard observability format, so this activity can flow into any compatible tool (Langfuse, Datadog, LangSmith, and others) with no vendor-specific code in Atryum.

Why it matters: there was previously no straightforward way to see, in an observability tool, what Atryum's governance layer was doing — which tool calls agents attempted, how the AI judge ruled, and what each judgment cost in time and tokens.

Before: governance activity was not exported anywhere.
After: each governed invocation produces a single trace showing:

• the tool call and its outcome — approved, denied, or sent for human approval;
• the AI-judge decision rendered as a generation: the model used, input/output token counts, and latency;
• the exact prompt the judge saw and the verdict it returned.

This works the same whether the judge runs inside Atryum (standalone / bring-your-own-key) or is hosted by the ValidMind backend — both paths render identically. Sending traces to a new backend is pure configuration (endpoint + credentials in atryum.toml); the same traces can fan out to several backends at once.

Configuration

Everything lives in one [otel] section of atryum.toml — no environment
variables, no code changes. Off by default. Providers are not hardcoded: any
OTLP backend works, and Langfuse/Datadog below are just examples.

Turn it on (or off)

[otel]
enabled = true   # omit the section or set false = no traces, fully no-op

Add a provider — one [[otel.exporters]] block per destination. Each is an
OTLP endpoint plus how to authenticate:

[[otel.exporters]]                                       # Basic-auth key pair (e.g. Langfuse)
name = "langfuse"                                        # a label, for logs only
endpoint = "https://<host>/api/public/otel/v1/traces"    # full URL, must end in /v1/traces
public_key = "pk-lf-…"
secret_key = "sk-lf-…"

[[otel.exporters]]                                       # API key in a header (e.g. Datadog)
name = "datadog"
endpoint = "https://otlp.<dd-site>/v1/traces"
headers = { "DD-API-KEY" = "<key>" }

Rule of thumb: auth goes in headers; public_key/secret_key is just a
shortcut for Basic-auth backends. List several blocks to fan the same traces to
several tools at once. Restart Atryum — the startup log prints otel tracing enabled.

Keep sensitive text out (optional)

[otel]
capture_content = false   # default true

When off, model / tokens / latency / verdict still export, but the judge's prompt
and answer text (which contain your charter/policy and the agent's tool arguments)
do not.

That's the whole surface: enabled, one or more [[otel.exporters]], and capture_content.

How it connects

Every governance decision Atryum makes is now emitted as a standard OpenTelemetry (OTEL) trace, so the same data can flow to any observability tool — we proved it reaching Langfuse and Datadog at the same
time, with no code change to add the second.

The shape of one trace

A single permission check produces one nested trace. Each layer is a "span":

Datadog

Langfuse

How to test

1. In atryum.toml, enable [otel] and add an exporter pointing at any OTLP-capable backend (endpoint + credentials).
2. Restart Atryum — the startup log confirms otel tracing enabled.
3. Drive a few invocations through the UI or the invocations API.
4. In your observability tool, filter for service:atryum: each invocation appears as a trace, and any AI-judged call shows a chat {model} generation with model, tokens, latency, and (unless content capture is disabled) the prompt and verdict.

Verified locally end-to-end against a real OTLP backend for both judge paths, including token counts, latency, and input/output.

What needs special review?

• internal/invocation/service.go — the VM-path generation span is synthesized after the fact and backdated by the reported latency so its duration is the real LLM call time; confirm the timestamps and nesting under atryum.ai_evaluation.
• The content-capture gate ([otel].capture_content, default on) — confirm it fully suppresses prompt/completion when off while still emitting model/tokens/latency/verdict.

Dependencies, breaking changes, and deployment notes

• Paired with a ValidMind backend change https://github.com/validmind/backend/pull/3390 that returns the judge model, token usage, latency, and messages from the evaluation endpoint, so backend-hosted judgments render fully. Atryum degrades gracefully without it (model-less span, no crash).
• No breaking changes. OTEL export is off unless [otel] is enabled; when disabled the tracer is a no-op.
• Deployment note: enabling export sends span data (and, unless capture_content = false, judge prompt/response text) to the configured backend.

Data impact

• Yes — this change is data-impacting
• No — this change is not data-impacting

Justification: Additive, opt-in telemetry plus a new optional config key ([otel].capture_content). No existing configuration, stored data, or API response is modified; export is inert unless [otel] is enabled.

Release notes

enhancement — Atryum can now emit its governance activity (tool calls and AI-judge decisions, including model, tokens, latency, and — optionally — prompt/response) as OpenTelemetry traces to any OTLP-compatible backend, configured per exporter in atryum.toml.

Checklist

• What and why
• Screenshots or videos (Frontend) — n/a, no UI change
• How to test
• What needs special review
• Dependencies, breaking changes, and deployment notes
• Labels applied
• Data impact classified
• PR linked to Shortcut
• Unit tests added
• Tested locally
• Documentation updated (if required)
• Environment variable additions/changes documented (if required)

[CLAassistant]

All committers have signed the CLA.

[mdeyell-valid-mind]

it's possible we could use something not from open ai in the future. Maybe we need a lookup table based off of model name?