[stewartshea]

What & why

RunWhen's self-hosted chart already deploys an optional OTLP collector + OpenObserve when runner.enabled is true (see deploy/scripts/registry-sync/sample_values.yaml and updated_values.yaml). The receive side exists; the producer side did not — nothing in the runner emitted OTLP, so the collector had nothing to receive.

This PR adds optional, off-by-default OTLP telemetry to RunWhen Local and wraps each run in a single span with sensor-style attributes, so a user can see what ran and how long it took.

⚠️ Assumption (please read)

The task described runwhen-local as "the local runner that executes RunWhen SLXs / codebundle tasks." In this repository, src/ is the RunWhen Local workspace builder — it indexes resources and renders workspace/SLX definitions; it does not execute codebundle tasks. That is the separate runner image referenced by runner.enabled in the chart (image runwhen/runner), which does not live in this repo.

Because that producer code is not present here, and because the ask says to keep the change minimal and to do the producer work rather than the chart/collector work, I instrumented the closest producer-side "run" this repo owns: the workspace-builder run (workspace_builder.run_handler.execute_run, the single entry point behind POST /run/). This is the narrowest sensible instrumentation available in this repository.

If the intent was to instrument the external runner image, this PR is not that — it is the producer-side change that fits this repo. I deliberately did not touch the chart's collector configuration (the receive side already exists).

Off-by-default contract

With no OTEL_* variables set:

• importing telemetry imports nothing from opentelemetry (covered by a test that runs import telemetry in a subprocess and asserts opentelemetry is not in sys.modules);
• no exporter is constructed and no network calls are made;
• the run path is unchanged — the span handle is an inert no-op.

Env vars (standard names only)

Variable	Effect
OTEL_TRACES_EXPORTER=otlp	enable OTLP trace export (any other value / unset = off)
OTEL_LOGS_EXPORTER=otlp	enable OTLP log export (any other value / unset = off)
OTEL_EXPORTER_OTLP_ENDPOINT	collector endpoint (_TRACES_ / _LOGS_ variants honored by the exporters)
OTEL_EXPORTER_OTLP_PROTOCOL	grpc (default) or http/protobuf
OTEL_SERVICE_NAME	service name (default runwhen-local)
OTEL_TRACES_SAMPLER / OTEL_TRACES_SAMPLER_ARG	sampler (read by the SDK)
OTEL_PROPAGATORS	only matters for cross-process context; this process is the root of each run, so the SDK default applies

What is instrumented

• src/telemetry.py (new) — lazy, best-effort OTLP setup (traces and, only if requested, logs). init_telemetry() is called on the startup path (workspace_builder/startup.py:bootstrap()); any failure logs a warning and continues without telemetry. Exporter construction sits behind _make_span_exporter() / _make_log_exporter() seams so tests can patch it with in-memory stubs (no network).

• src/workspace_builder/run_handler.py — execute_run now wraps the run in one span named runwhen.workspace_builder.run with attributes:

  • runwhen.workspace_name, runwhen.location_id, runwhen.location_name
  • runwhen.components, runwhen.namespaces
  • runwhen.status (ok / error), runwhen.slx_count, runwhen.warnings_count
  • OTel span status OK/ERROR; exceptions recorded via record_exception.

  The original body is unchanged apart from being renamed to _execute_run_pipeline(request_data, run_span); behaviour with telemetry off is byte-identical. This is one span per run — name/attributes are sensor-style, and duration is the span duration.

Dependencies

opentelemetry-api, opentelemetry-sdk, opentelemetry-exporter-otlp-proto-grpc, opentelemetry-exporter-otlp-proto-http added to src/pyproject.toml; src/poetry.lock regenerated with the Poetry version pinned in src/Dockerfile (2.5.1). poetry check --lock passes.

How I tested it

• New src/test_telemetry.py (12 tests), hermetic (exporters patched with an in-memory exporter / failing stub, no network):
  • nothing configured → no exporter built, get_tracer() is None, span handle inert, no crash;
  • traces enabled → exactly one span with the expected name and attributes (and OTel status);
  • broken exporter → warning logged, no exception, run continues;
  • import telemetry in a fresh subprocess does not import opentelemetry;
  • the execute_run wrapper is transparent when disabled and records ok/error spans otherwise.
• Full unit suite in src/: 506 passed, 4 subtests passed, 0 failed (python -m pytest -q).

How to use it with the existing collector

Set the standard env vars on the container (e.g. via the chart's runwhenLocal env / extraEnv):

OTEL_TRACES_EXPORTER=otlp
OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4317
OTEL_EXPORTER_OTLP_PROTOCOL=grpc
OTEL_SERVICE_NAME=runwhen-local

Deliberately left out

• The chart's collector configuration (deploy/scripts/registry-sync/*values.yaml) — the receive side already exists.
• The external runner image (not in this repo) — see the assumption above.
• OTEL_PROPAGATORS handling — only relevant for cross-process propagation, and this process is the root of each run.
• Metrics — the ask is spans (plus optional logs); metrics are out of scope.