Skip to content

perf(telemetry): spool events locally and ship them from a detached process - #42

Merged
igor-ctrl merged 3 commits into
mainfrom
feat/telemetry-spool
Oct 30, 2026
Merged

igor-ctrl merged 3 commits into
mainfrom
feat/telemetry-spool

Conversation

@igor-ctrl

@igor-ctrl igor-ctrl commented Oct 30, 2026 •

Copy link
Copy Markdown
Owner

Problem

The azure_monitor telemetry backend went through azure-monitor-opentelemetry. In a short-lived CLI that costs a lot:

  • ~2 s added to every command. The exporter shuts down synchronously at process exit; bcli endpoint list went from 0.3 s to 2.3 s. Disabling tracing, metrics and live metrics didn't change it.
  • A traceback on every command on any machine that isn't an Azure VM. The Azure VM resource detector probes the instance metadata endpoint (IMDS) and logs the timeout with a full traceback (~70 lines of stderr).

Change

  • emit() appends one Application Insights envelope per event to ~/.config/bcli/telemetry/spool.jsonl. It takes microseconds and makes no network call.
  • flush() (called once at exit) starts a detached python -m bcli.telemetry._ship (start_new_session on POSIX, DETACHED_PROCESS on Windows) that posts the spool to the connection string's IngestionEndpoint with httpx, after the command has returned.
  • The shipper:
    • holds an O_EXCL lock, and takes it over once stale (5 min);
    • claims the spool by renaming it, so concurrent commands start a fresh spool;
    • posts in batches of 500 and drops unparseable lines rather than losing the whole batch to a 400;
    • keeps the unsent remainder on a network error or a retryable status (408/429/439/5xx).
  • The spool is capped at 5 MB, so an offline machine can't grow it without bound.
  • The [telemetry] extra is now empty and kept so existing install commands resolve, mirroring cli = []. The lockfile drops the OpenTelemetry/Azure Monitor dependency tree.
  • No config change. Same backend = "azure_monitor", same connection_string.

Events now arrive as real custom events (customEvents / AppEvents) instead of traces. Properties are string-valued, so numbers read back with todouble(...).

Verification

  • bcli endpoint list against a real Application Insights resource: 0.29–0.32 s with telemetry on vs 0.28–0.29 s off, nothing on stderr beyond normal output. Previously it took 2.3 s and printed the traceback.
  • All events from 5 runs (5 × bcli.startup, 5 × bcli.command) arrived in AppEvents, and the spool directory was empty afterwards.
  • New tests/test_telemetry/test_ship.py covers: batching; retry on 503/429/connection error with only the remainder kept; final 400 not retried; malformed lines dropped; fresh lock respected; stale lock taken over.
    • Mutation-checked: treating every response as final fails 2 tests, and ignoring lock age fails 1.
  • test_sink.py covers: the envelope shape, connection-string parsing (including the default endpoint), the spool cap, an unwritable spool never raising, and flush() spawning only after something was spooled.
  • Full suite: 1221 passed, 5 skipped. ruff check src/ is clean.

Not exercised: the Windows DETACHED_PROCESS path on a real Windows machine.


Note

Medium Risk
Changes how optional telemetry is persisted on disk and uploaded (background subprocess, spool retention), though it does not touch auth or core API paths.

Overview
Replaces the azure_monitor backend’s azure-monitor-opentelemetry integration with local spooling and a detached shipper, so CLI commands no longer pay ~2s exit-time exporter shutdown or Azure VM IMDS probe noise on non-Azure machines.

During a command, emit() appends Application Insights event envelopes to ~/.config/bcli/telemetry/spool.jsonl (string-valued properties, 5 MB cap with drops past the limit). flush() at exit starts python -m bcli.telemetry._ship, which posts batches via httpx, uses a single-shipper lock (stale takeover), renames the live spool to outbox files for concurrent runs, and retains unsent lines on retryable failures.

The [telemetry] extra is a no-op ( httpx is core); lockfile drops the OpenTelemetry/Azure Monitor tree. No config change — same backend and connection_string. Tests cover the shipper and updated sink behavior.

Reviewed by Cursor Bugbot for commit de3df00. Bugbot is set up for automated code reviews on this repo. Configure here.

…rocess

The azure_monitor backend went through azure-monitor-opentelemetry, whose
exporter shut down synchronously at process exit. That added ~2 s to every
command (0.3 s -> 2.3 s for `bcli endpoint list`), and its Azure VM resource
detector printed an IMDS timeout traceback on any machine that isn't an
Azure VM.

emit() now appends one Application Insights envelope per event to
~/.config/bcli/telemetry/spool.jsonl. flush(), called once at exit, starts
a detached `python -m bcli.telemetry._ship` that posts the spool to the
connection string's ingestion endpoint with httpx, after the command has
returned. Commands run in the same time with telemetry on as with it off.

The shipper holds an O_EXCL lock (taken over once stale), claims the spool
by renaming it, posts in batches, deletes what Application Insights has
answered for, and keeps the unsent remainder on a network error or a
retryable status. The spool is capped at 5 MB.

Events now arrive as custom events (AppEvents) with string-valued
properties. The [telemetry] extra is kept empty so existing install
commands still resolve.

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit d578414. Configure here.

Comment thread src/bcli/telemetry/_azure_monitor.py Resolved
At the 5 MB cap emit() dropped the event without marking work pending, so
flush() never started the shipper and a full spool could never drain:
telemetry stopped for good on that machine even after it came back online.
Hitting the cap now marks the backlog pending like a successful write does.
@igor-ctrl
igor-ctrl merged commit a1f41e5 into main Oct 30, 2026
4 checks passed
@linear-code

linear-code Bot commented Oct 30, 2026

Copy link
Copy Markdown

TECH-1279

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant