Skip to content

[miniflare] Capture local observability spans over OTLP - #16204

Draft
danlapid wants to merge 1 commit into
cloudflare:mainfrom
danlapid:dlapid/observability-otlp
Draft

danlapid wants to merge 1 commit into
cloudflare:mainfrom
danlapid:dlapid/observability-otlp

Conversation

@danlapid

@danlapid danlapid commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

Depends on cloudflare/workerd#7686.

Spans are going away from streaming tail events. cloudflare/workerd#7686 lets workerd export each invocation's spans itself, as OTLP, through a new tracing.otlp config setting. This moves Miniflare's local observability (unsafeObservability) onto it:

  • Spans over OTLP. Miniflare sets tracing.otlp to the observability collector. Its new POST /v1/traces decodes workerd's protobuf export (a small reader for the fields workerd sets, no new dependency) and stores the spans. They carry precise timestamps instead of the coarse I/O time tail events report.
  • Logs stay on the streaming tail. The tail handler now handles only logs, exceptions, and the per-invocation log. tail_worker_user_spans is no longer enabled. Logs still reference spans correctly: the tail events carry the same span IDs workerd exports.
  • Only the user's spans are kept. tracing.otlp traces every service in the config, including Miniflare's own Workers (the entry Worker, simulators, queue brokers, …). The collector receives a map from workerd service name to worker name: user workers, plus the Workflows services that run their workflows. It keeps only those services' spans. This map replaces the per-tail props that used to carry the worker name.
  • Roots stay roots. Dropping Miniflare's spans would leave, for example, an invocation that came through the entry Worker pointing at a parent that is never stored. The store keeps a span whose parent is missing as a root and parks the parent in a pending_parents table. If the parent arrives later, the span is attached to it again. That happens when one user worker calls another, because the callee usually finishes and is exported first. So distributed traces across user workers still join up.

The config types in src/runtime/config/generated/workerd.ts are regenerated from the workerd schema in #7686 (the only difference is the new tracing field). This can't merge until a workerd release that includes #7686 is published and Miniflare's workerd dependency is bumped: with today's workerd, tracing is ignored and no spans are captured. Until then, CI fails the tests that need captured spans: the four capture tests in test/plugins/core/observability.spec.ts and "captures the selected same-instance Worker in Observability" in test/plugins/local-explorer/scheduled.spec.ts.

While tracing.otlp is set, workerd stops delivering spans to streaming tail workers, so with local observability on (the default) a streaming tail worker configured locally no longer receives spanOpen/spanClose. That's intended: spans on streaming tail workers are being deprecated in local dev and production alike, and users can't deploy streaming tail workers.

A failed span's error column is its last exception (workerd records uncaught exceptions on the invocation span), else its status message, else its error.type, so a span that failed without an exception, such as a 5xx fetch, still counts as an error.


  • Tests
    • Tests included/updated
    • Automated tests not possible - manual testing has been completed as follows:
    • Additional testing not necessary because:
  • Public documentation
    • Cloudflare docs PR(s):
    • Documentation not necessary because: unsafeObservability is an unsafe, undocumented option, and what the Local Explorer shows is unchanged.

Tested against workerd built from #7686 (MINIFLARE_WORKERD_PATH): test/plugins/core/observability.spec.ts, test/plugins/local-explorer/, test/plugins/workflows/, test/plugins/core/ and test/index.spec.ts all pass. New tests cover parent re-linking in the store and dropping Miniflare's own spans.

🤖 Generated with Claude Code

Spans are going away from streaming tail events: workerd can now export
each invocation's spans itself, as OTLP (`tracing.otlp`, from
cloudflare/workerd#7686). With `unsafeObservability`, Miniflare makes the
observability collector workerd's OTLP collector. Its new `/v1/traces`
endpoint decodes the protobuf export with a small reader for the fields
workerd sets and stores the spans; the streaming tail now supplies only
logs and exceptions, so `tail_worker_user_spans` is no longer enabled.

`tracing.otlp` traces every service in the config, including
Miniflare's own Workers. The collector keeps only the spans of the
services Miniflare passes it (user workers and the Workflows services
that run their workflows, with the worker names the tail props used to
carry) and drops the rest. An invocation started by one of those
dropped spans, such as the entry Worker's, would then point at a parent
that is never stored, so the store keeps a span whose parent is missing
as a root and parks the parent in `pending_parents`. If the parent
arrives later, as a calling user worker's span does after its callee
has been exported, the span is attached to it again.

The config types are regenerated from the workerd schema that adds
`tracing`, so this needs a workerd release that includes it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@changeset-bot

changeset-bot Bot commented Oct 11, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: f3f9d28

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 8 packages
Name Type
miniflare Minor
@cloudflare/deploy-helpers Patch
@cloudflare/pages-shared Patch
@cloudflare/remote-bindings Patch
@cloudflare/runtime-types Patch
@cloudflare/vite-plugin Patch
@cloudflare/vitest-plugin Patch
wrangler Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@pkg-pr-new

pkg-pr-new Bot commented Oct 11, 2026

Copy link
Copy Markdown
@cloudflare/autoconfig

npm i https://pkg.pr.new/@cloudflare/autoconfig@16204

@cloudflare/build-output-utils

npm i https://pkg.pr.new/@cloudflare/build-output-utils@16204

@cloudflare/codemods

npm i https://pkg.pr.new/@cloudflare/codemods@16204

@cloudflare/config

npm i https://pkg.pr.new/@cloudflare/config@16204

@cloudflare/containers-shared

npm i https://pkg.pr.new/@cloudflare/containers-shared@16204

create-cloudflare

npm i https://pkg.pr.new/create-cloudflare@16204

@cloudflare/deploy-helpers

npm i https://pkg.pr.new/@cloudflare/deploy-helpers@16204

@cloudflare/kv-asset-handler

npm i https://pkg.pr.new/@cloudflare/kv-asset-handler@16204

miniflare

npm i https://pkg.pr.new/miniflare@16204

@cloudflare/pages-functions

npm i https://pkg.pr.new/@cloudflare/pages-functions@16204

@cloudflare/pages-shared

npm i https://pkg.pr.new/@cloudflare/pages-shared@16204

@cloudflare/runtime-types

npm i https://pkg.pr.new/@cloudflare/runtime-types@16204

@cloudflare/unenv-preset

npm i https://pkg.pr.new/@cloudflare/unenv-preset@16204

@cloudflare/vite-plugin

npm i https://pkg.pr.new/@cloudflare/vite-plugin@16204

@cloudflare/vitest-plugin

npm i https://pkg.pr.new/@cloudflare/vitest-plugin@16204

@cloudflare/workers-auth

npm i https://pkg.pr.new/@cloudflare/workers-auth@16204

@cloudflare/workers-editor-shared

npm i https://pkg.pr.new/@cloudflare/workers-editor-shared@16204

@cloudflare/workers-utils

npm i https://pkg.pr.new/@cloudflare/workers-utils@16204

wrangler

npm i https://pkg.pr.new/wrangler@16204

commit: f3f9d28

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Untriaged

Development

Successfully merging this pull request may close these issues.

2 participants