feat(agent-bff): bundle opentelemetry apm in the docker image

[nbouliol]

Stacked on #1848 — review that one first, this PR's diff is only the OpenTelemetry layer.

Bundles OpenTelemetry APM into the agent-bff Docker image, the way the workflow-executor image already does.

How it works

src/tracing.ts holds the setup; src/tracing-preload.ts is the two-line --require entry point the Dockerfile's ENTRYPOINT loads before cli.js, so the auto-instrumentation patches land before the app imports anything. Keeping them apart is what lets the setup be imported by a test without arming an SDK.

Tracing is off until OTEL_EXPORTER_OTLP_ENDPOINT is set, so a deployment that never opted into APM pays nothing but the require. Configuration is entirely standard OTel environment variables (OTEL_SERVICE_NAME, OTEL_RESOURCE_ATTRIBUTES, OTEL_SDK_DISABLED).

The packages are pinned in the image's isolated dependency tree only, never shipped to npm consumers — hence the dynamic require behind a seam, and a warning rather than a crash when they are absent.

Two things worth a look

The signal handler re-raises. The BFF installs no signal handler of its own, and registering a listener suppresses Node's default termination — arming one to flush spans without re-raising would leave docker stop waiting out its timeout for a SIGKILL. So the handler flushes, then re-raises the same signal; once has already removed the listener, so the second delivery finds none and terminates. A rejected flush re-raises anyway, so a dead collector cannot hang the process.

scan-gate.js blocks again. @opentelemetry/* exists only in this image and can only be fixed here, so a fixable CRITICAL/HIGH in one of them now fails the publish. Everything else stays report-only. **/@opentelemetry/propagator-jaeger is pinned to 2.9.0 (CVE-2026-59892) — auto-instrumentations-node still asks for 2.8.0.

Verified locally (arm64)

• image rebuilt, smoke test passes and now asserts OpenTelemetry tracing enabled in the logs — the packages exist only in the image, so nothing else would prove they are loadable
• yarn audit on the regenerated isolated lockfile: 0 vulnerabilities across 335 packages
• 11 new tests on the setup (both load and signal paths through injected seams); the full agent-bff suite is green at 1217 tests
• lint clean

🤖 Generated with Claude Code

Note

Bundle OpenTelemetry APM into agent-bff Docker image

• Adds tracing-preload.ts loaded via node --require in the Dockerfile ENTRYPOINT so auto-instrumentation is applied before app modules load. Initialization is gated on OTEL_EXPORTER_OTLP_ENDPOINT and stays inert otherwise.
• Adds tracing.ts with environment-driven SDK setup, OTLP exporter selection, credential redaction in logs, and a dynamic module loader that tolerates missing OTel packages outside Docker.
• Extends shutdown.ts to run a bounded telemetry flush in parallel with server stop; flush failures do not affect exit codes.
• Updates build-deps-manifest.js to merge pinned @opentelemetry/* packages into the Docker dependency install, and scan-gate.js now fails the image publish on vulnerabilities found in Docker-only @opentelemetry/* packages.
• Risk: scan-gate.js now calls process.exit(1) on vulnerabilities in @opentelemetry/* packages; bumping versions in OTEL_DEPENDENCIES requires refreshing deps/yarn.lock.

Changes since #1849 opened

• Changed tracing enablement logic in initTracing to activate when either OTEL_EXPORTER_OTLP_ENDPOINT or OTEL_TRACES_EXPORTER environment variables contain non-blank values, treating blank values as unset, while continuing to respect OTEL_SDK_DISABLED as a short-circuit [296a6c2]
• Wrapped setTracingHandle(initTracing()) call in tracing-preload.ts with try-catch to prevent exceptions during --require preload from aborting the process, logging a warning and continuing untraced on failure [296a6c2]
• Introduced serviceNameFromEnv utility and modified initTracing to respect OpenTelemetry specification precedence for service name resolution, only applying default when environment provides no name [296a6c2]
• Added validation in loadOtelModules to verify that resolved OpenTelemetry packages export the expected functions (NodeSDK, getNodeAutoInstrumentations, OTLPTraceExporter), returning undefined if any export is missing or non-function [296a6c2]
• Enhanced smoke test in docker/smoke-test.sh to enable tracing via OTEL_EXPORTER_OTLP_ENDPOINT and validate graceful shutdown within bounded time when collector is unreachable [296a6c2]
• Expanded test coverage in tracing.test.ts and tracing-preload.test.ts to verify new enablement conditions, service name precedence, module export validation, and error handling during initialization [296a6c2]
• Removed OTEL_DEPENDENCIES export from docker/build-deps-manifest.js module and extended comments explaining pinned OpenTelemetry dependency policy and Renovate constraints [296a6c2]
• Updated README.md documentation to clarify OpenTelemetry tracing behavior including enablement conditions, service name precedence, SDK disable semantics, shipped exporters, and failure handling [296a6c2]
• Removed manual construction of OTLPTraceExporter from @opentelemetry/exporter-trace-otlp-http in the agent-bff tracing initialization [5a07686]
• Documented OpenTelemetry environment variable configuration in agent-bff README [5a07686]
• Updated agent-bff tracing test suite to remove OTLPTraceExporter expectations and validate SDK-based configuration [5a07686]
• Expanded documentation explaining OpenTelemetry package bundling behavior in Docker image [50d3f1d]

^{Macroscope summarized d5a1f19.}

[qltysh]

3 new issues

Tool	Category	Rule	Count
qlty	Structure	Function with high complexity (count = 12): generate	3

[qltysh]

Coverage Impact

This PR will not change total coverage.

Modified Files with Diff Coverage (4)

Rating	File	% Diff	Uncovered Line #s
	packages/agent-bff/src/shutdown.ts	100.0%
	packages/agent-bff/src/tracing.ts	100.0%
	packages/agent-bff/src/tracing-preload.ts	100.0%
	packages/agent-bff/src/tracing-handle.ts	100.0%
	Total	100.0%

🚦 See full report on Qlty Cloud »

🛟 Help

• Diff Coverage: Coverage for added or modified lines of code (excludes deleted files). Learn more.

• Total Coverage: Coverage for the whole repository, calculated as the sum of all File Coverage. Learn more.

• File Coverage: Covered Lines divided by Covered Lines plus Missed Lines. (Excludes non-executable lines including blank lines and comments.)

  • Indirect Changes: Changes to File Coverage for files that were not modified in this PR. Learn more.

[Tonours]

Good layer, and the design is better than the description says (more on that below). Splitting tracing.ts (pure setup) from tracing-preload.ts (the --require) from tracing-handle.ts (the one shared bit of state) is the right split, and wiring the flush into armShutdown instead of letting the preload arm its own handler is what makes the exit wait for the export rather than race it. 19 tests on shutdown.ts covering the flush deadline, a failing stop() with a flush still in flight, and the interrupted paths.

I built the image and exercised it rather than take the claims on trust. Built from d5a1f197b, arm64.

Check	Result
docker build	exit 0, 337 MB, ENTRYPOINT node --require .../tracing-preload.js .../cli.js
OTel packages shipped	79 under node_modules/@opentelemetry, node_modules 36 MB -> 152 MB
smoke-test.sh	exit 0
Tracing off, no endpoint	no tracing line, docker stop in 0s
Tracing on, collector unreachable	OpenTelemetry tracing enabled, docker stop in 3s, exit 0
Flush deadline	Shutting down -> Forest BFF stopped = 2.006s, exactly flushMs
OTEL_SDK_DISABLED=TRUE	disabled, the uppercase fix holds
Credential redaction	https://user:s3cret@collector.example:4318 logged as https://collector.example:4318/, zero occurrences of the secret

So the bounded() deadline does exactly what it promises: a dead collector costs 2 seconds, not the 10 the in-flight requests get, and the process still ends on 0. The "~3 seconds" in the README is the real number.

Two things on the description itself, since neither is a line in the diff:

The "signal handler re-raises" section is stale. It describes once plus a re-raise of the same signal, and none of that is in the final code: 4a087d940 and 6a8bf5cae moved the flush into armShutdown through tracing-handle, and tracing.ts:13-14 now says arming a handler there would be the bug. Worth rewriting before merge, it's the part reviewers will read, and it points them at a mechanism that no longer exists.

The size cost isn't mentioned. 271 MB -> 337 MB, +24%, and node_modules more than quadruples, for a layer that is off by default. Nothing to fix, auto-instrumentations-node pulls the 79 packages and that's the price. But #1848 put "image 271 MB" in its description, so this one saying nothing means someone finds the number in production instead.

Line comments below. Nothing blocking, one is worth three lines of guard before merge.

[Tonours]

Approving. I went back through all 16 findings and checked each one against the code at 50d3f1d, not against the replies.

What I verified:

• Signal handling — tracing.ts arms nothing, the SDK is parked on tracing-handle and armShutdown is the only terminating handler. Pinned by tracing.test.ts:239.
• Credentials in the logs — redactEndpoint (tracing.ts:121), applied at the log line. Pinned, user:password and token-as-user.
• OTEL_SDK_DISABLED — trim + lowercase, pinned over TRUE / True / ' true '.
• Promise.all on the shutdown path — now allSettled with the server result read out of it. Pinned by shutdown.test.ts:118.
• Module that resolves without the export we read — every export checked callable, plus the preload catching whatever is left. Pinned per package.
• Exporter, protocol and per-signal endpoint — root cause removed rather than patched per variable: no traceExporter at all, the SDK reads the environment. Pinned.
• The flush path in the image — smoke-test.sh:160-214 arms OTLP against a dead collector on the second boot and asserts exit 0 with a bounded elapsed. Confirmed it runs in CI on this head (docker-publish-bff run 32994136258, success), so this and the scan gate are actually exercised.

Two things I did not ask you to change: openapi not flushing is the right call at 9s versus 1.3s, and the options.env seam stopping short of where spans go is the honest boundary — both are documented where someone will hit them.

One nit, not blocking: the fix for the OTEL_TRACES_EXPORTER gate left three comments behind saying tracing only arms on OTEL_EXPORTER_OTLP_ENDPOINT — Dockerfile:114, tracing.ts:10, and the variable list at tracing.ts:15-20 which omits OTEL_TRACES_EXPORTER and OTEL_EXPORTER_OTLP_TRACES_ENDPOINT. The README table is correct; these are not.

Note for merge order: this stacks on feat/agent-bff-docker-image, so the parent goes first.