feat(observe): export settings from the standard OpenTelemetry variables, and an event time on log records

[arun-pathiban-ddog]

What and why

temper serve can export traces, metrics and logs over OTLP/HTTP. Running it as one of many server processes, against an OTLP backend that is stricter than a typical collector, showed four gaps. This closes them using the standard OpenTelemetry environment variables. With none of the new variables set, the startup output and the export are unchanged apart from the event time on log records.

1. Log records carried no event time. The log bridge sets only the observed time, so records were exported with an event time of zero. A backend that dates records by event time treats them as very old and can drop them while still answering with success. A log processor placed before the batch processor now sets the event time to the observed time when a record has none.
2. Every server process reported the same identity. OTEL_SERVICE_NAME now replaces the built-in service name and OTEL_RESOURCE_ATTRIBUTES is merged into the resource of all three signals.
3. Signals could not be switched off one by one. OTEL_TRACES_EXPORTER, OTEL_METRICS_EXPORTER and OTEL_LOGS_EXPORTER set to none now switch that signal's exporter off.
4. Sampling always followed the caller. OTEL_TRACES_SAMPLER now chooses the sampler. The name-based filter stays around whichever sampler is chosen.

All changes are in crates/temper-observe. The call site in temper serve did not need to change: it still calls init_observability("temper-platform").

New variables

Variable	Default	Effect
OTEL_SERVICE_NAME	the built-in name (temper-platform for temper serve)	Service name on traces, metrics and logs.
OTEL_RESOURCE_ATTRIBUTES	none	Extra resource attributes as key=value,key=value on traces, metrics and logs. Before this change only deployment.environment.name was read from it.
OTEL_TRACES_EXPORTER	otlp	none switches the export of traces off.
OTEL_METRICS_EXPORTER	otlp	none switches the export of metrics off.
OTEL_LOGS_EXPORTER	otlp	none switches the export of logs off.
OTEL_TRACES_SAMPLER	parentbased_always_on	always_on, always_off, traceidratio, parentbased_always_on, parentbased_always_off or parentbased_traceidratio.
OTEL_TRACES_SAMPLER_ARG	1	Ratio from 0 to 1 for the two ratio samplers.

Behaviour worth a reviewer's attention:

• Precedence. What the server computes itself wins over OTEL_RESOURCE_ATTRIBUTES: runtime-id always, deployment.environment.name when DD_ENV or LOGFIRE_ENVIRONMENT is set, service.version when DD_VERSION is set. A service.name inside OTEL_RESOURCE_ATTRIBUTES does not replace the built-in name; only OTEL_SERVICE_NAME does. The OpenTelemetry specification would let it; this is deliberate, so that a deployment that already carries it there keeps its service name.
• A signal that is switched off keeps its provider and gets no exporter. Spans still carry trace context and log lines are still printed.
• Bad values never stop the server. A value that is not supported is logged once at startup as a warning, and the default applies. A bad ratio falls back to the ratio's default, 1, with the chosen sampler kept, as the OpenTelemetry specification says. Entries of OTEL_RESOURCE_ATTRIBUTES that are not key=value are counted in the warning and not printed.
• OTEL_TRACES_SAMPLER now takes effect. Until now it was ignored, because the setup always installed its own sampler. A deployment that already has it in its environment will see it applied.
• Sampling tests use a span whose remote parent is marked "not sampled". The server's own request path (remote_parent_context) marks every remote parent as sampled, and that is unchanged here.

Not done

• Reporting rejected records. An OTLP backend can answer with success and still report rejected records in the response body. The pinned exporter (opentelemetry-otlp 0.28.0) checks only the status code and drops the body, in src/exporter/http/{trace,metrics,logs}.rs, so there is nothing to report from. Doing it would mean wrapping the exporter's HTTP client and decoding the response here, with new direct dependencies. Left out.
• The suspected dropped-span defect was checked and does not apply. With per-layer filters, asking the log bridge "is logging enabled?" without logging can leave filter state on the thread, and the next span on that thread is dropped. The subscriber here filters globally, so it is not affected. The test is kept as a guard: it fails if the same layers are given per-layer filters.

How it is tested

crates/temper-observe/tests/otel_export starts the real setup (init_observability) in a child process for each case, with a clean environment plus the variables under test, pointed at a local OTLP/HTTP listener that records every request. The tests assert on the decoded requests and on what the child printed. No protobuf dependency is added; the tests read the few fields they need from the wire format.

default_export.baseline.txt is the startup output and the full export with no new variable set. It was recorded in the first commit, before any change to the setup. The only change to it in this pull request is the event time of each log record (time=<zero> to time=<observed time>).

Acceptance checks

All commands were run locally on the final commit of this branch. cargo test -p temper-observe runs 75 unit tests and the 24 tests of otel_export; all pass (cargo nextest run -p temper-observe: 99 run, 99 passed).

#	Criterion	Status	Evidence
1	Every exported log record has a non-zero event time; the test fails first because the event time is zero	passed	every_log_record_has_an_event_time. Before the fix: log record "test log inside a span" was exported with an event time of zero, left: 0, right: 0. After it: passes. Against a running temper serve: 466 log records, 0 with an event time of zero.
2	OTEL_SERVICE_NAME=x gives service.name = x; unset gives the built-in name	passed	identity::service_name_follows_otel_service_name, identity::service_name_is_built_in_when_unset, on all three signals.
3	OTEL_RESOURCE_ATTRIBUTES=a=1,b=2 is on traces, metrics and logs; runtime-id still present; computed attributes not overridden	passed	identity::resource_attributes_are_merged_on_every_signal, identity::computed_resource_attributes_keep_precedence, identity::resource_attributes_supply_what_has_no_variable_of_its_own.
4	OTEL_METRICS_EXPORTER=none: traces and logs arrive, no metrics request; same for the other two signals	passed	signals::metrics_exporter_none_switches_only_metrics_off, signals::traces_exporter_none_switches_only_traces_off, signals::logs_exporter_none_switches_only_logs_off. Against a running temper serve with OTEL_METRICS_EXPORTER=none: 3 trace requests, 3 log requests and no metrics request in 40 s (the metric interval is 30 s).
5	OTEL_TRACES_SAMPLER=always_on: a request marked "not sampled" still produces a span with the caller's trace ID; unset, it produces none	passed	sampling::always_on_records_a_request_marked_not_sampled, sampling::default_sampler_follows_a_caller_that_did_not_sample. Tested with a span whose remote parent is marked not sampled; see the note on the server's request path above.
6	traceidratio with argument 0 produces no spans, with 1 all	passed	sampling::traceidratio_zero_exports_no_spans, sampling::traceidratio_one_exports_every_span.
7	The name-based filter drops what it drops today under every sampler	passed	sampling::every_sampler_keeps_the_name_based_filter: nine sampler and ratio combinations; under each, clock_time_get, turso.configured_connection and the reduced-rate name on a trace ID outside the rate are not exported. The existing unit tests of the filter are unchanged and pass.
8	An unknown value in any new variable produces one startup warning and default behaviour	passed	bad_values::unsupported_exporter_is_reported_once_and_exported (all three variables), bad_values::unsupported_sampler_is_reported_once_and_the_default_is_used, bad_values::bad_sampler_ratio_is_reported_once_and_one_is_used, bad_values::malformed_resource_attributes_are_reported_once_and_ignored, bad_values::empty_values_are_the_same_as_unset. OTEL_SERVICE_NAME takes any non-empty text, so it has no unknown value.
9	With no new variable set, a recorded export is identical to one from before the change, apart from the event time on logs	passed	default_export_matches_recorded_baseline. The baseline was recorded in the first commit, before any change; its only diff in this pull request is time=<zero> to time=<observed time> on the four log records.
10	A rejected-records response produces one warning, or the reason the SDK does not allow it is recorded	passed (reason recorded)	Not built. See "Not done": the pinned exporter drops the response body on success.
11	The probe-then-span test exists and passes	passed	log_enabled_probe_does_not_drop_the_next_span. It fails when the layers are given per-layer filters, with test.request missing from the export.
12	Workspace tests, lints and format checks show nothing new; CI is green	passed locally; CI not yet run	Locally: cargo fmt --check, cargo check --workspace, cargo clippy --workspace --all-targets -- -D warnings, the readability ratchet, the storage dispatch boundary check, the TODO and unwrap() scans, check_instrumentation and cargo test --doc --workspace all pass. cargo nextest run --workspace -E 'not test(dst_)', the command CI uses: 3505 tests run, 3505 passed, 65 skipped. CI has not run yet.
13	The observability documentation lists the new variables and their defaults	passed	The module documentation of temper_observe::otel and "Telemetry export settings" in Appendix C of docs/AGENT_GUIDE.md.

Also covered, from the constraints: bad_values::credentials_are_sent_and_never_logged (the credential in OTEL_EXPORTER_OTLP_HEADERS reaches the listener on every signal and appears in no output), and a run of the built temper serve with OTEL_SERVICE_NAME, OTEL_RESOURCE_ATTRIBUTES, OTEL_METRICS_EXPORTER=none, OTEL_TRACES_SAMPLER=always_on and an unsupported OTEL_LOGS_EXPORTER: it started, logged one warning, and exported traces and logs under the chosen name and attributes.

Lint baseline against the result

Check	main at f66287d	This branch
cargo fmt --check	clean	clean
cargo clippy --workspace --all-targets -- -D warnings	0 warnings	0 warnings
Readability ratchet, blocking metrics	pass	pass, every blocking metric unchanged (PROD_PRINTLN_COUNT 245, PROD_FILES_GT500 78)
Readability ratchet, advisory PROD_FILES_GT300	253	253

Departures from the plan

• Warnings go through tracing::warn! after the subscriber is installed, not through eprintln! like the existing endpoint messages. They are then exported as log records too, and the readability ratchet's println count does not grow.
• One new dev-dependency: log, in temper-observe, to make the "is logging enabled?" probe through the real log bridge. It was already in the lock file as a transitive dependency; Cargo.lock gains one line.
• The baseline for "unchanged by default" is recorded through the telemetry setup's entry point in a child process, not from a running temper serve. It is the same function temper serve calls, with the same service name.
• The bad-value tests were written after the code they cover, since each variable's handling of bad values came with the step that added the variable. Every other test in the series was written first and seen to fail for the stated reason.

🤖 Generated with Claude Code

Confidence Score: 5/5

The PR appears safe to merge; no actionable issue was established in the changed code.

Summary

The PR adds OpenTelemetry environment-based resource identity, per-signal export controls, and sampler selection, and fills in missing event times on exported logs.

• Adds unit and child-process export tests for defaults, overrides, bad values, and disabled signals.
• Documents the new settings and their precedence.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  Env[OTEL environment variables] --> Settings[ExportSettings]
  Settings --> Resource[Shared resource]
  Settings --> Sampler[Name-based filter and chosen sampler]
  Settings --> Signals{Signal enabled?}
  Sampler --> Traces[Tracer provider]
  Resource --> Traces
  Resource --> Metrics[Meter provider]
  Resource --> Logs[Logger provider]
  Signals -->|yes| Exporters[OTLP exporters]
  Logs --> EventTime[Fill missing event time]
  EventTime --> Exporters

Loading

_{Reviews (1) · Last reviewed commit: "docs(observe): list the telemetry export..."}