Skip to content

feat(tracing): OTLP span export to Langfuse and CloudWatch (phase 2, #1643) - #35792

Merged
bolichen97 merged 1 commit into
mainfrom
crew/devmate/otel-pr2
Nov 12, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
crew/devmate/otel-pr2

Conversation

@kyleseaman

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Kiro Crew can send its agent-turn traces to Langfuse, AWS CloudWatch or any OTLP collector the operator names, with auth headers kept in the vault.

Phases 1a (#31460) and 1b (#35077) record one trace per agent turn, sub-agents included, but only to local JSONL files under ~/.kiro/crew/traces. The customer ask in #1643 is to see those traces, with model, credits, tokens and tool calls, in their own trace backend. Today nothing leaves the host, so a Langfuse or CloudWatch user sees nothing.

Why it matters

This is the phase of the native tracing RFC (#31433, docs/request-for-change/rfc-otel-tracing.md, phase 2 row of §10) that a customer actually sees. Without it the traces recorded since 1a are only useful to someone reading JSONL on the machine.

Not a goal

  • Native SigV4 signing for CloudWatch (RFC phase 2b). CloudWatch works here through an X-Ray API key or a SigV4 collector.
  • A Settings UI for traces (RFC phase 7). Configuration is config.json and the vault.
  • Prompt, completion or tool content on spans (RFC phase 9, under Sensitive Change Review).
  • Changing what the metrics signal exports.

What changed (motivation → approach → change)

Goal → approach. Reuse what metrics already trusts for egress rather than adding a second path: the same otlp extra, the capabilities.telemetry otlp_hosts policy with its per-export recheck and audit row, the LTS lock, and main's no-redirect session.

What you can now do. With telemetry.traces_enabled on, set telemetry.traces_otlp_endpoint to a collector's /v1/traces URL, store the auth header in Settings → Secrets, and map it in traces_otlp_headers_secret. Each turn then reaches that collector. Langfuse shows each chat span as a generation with its Kiro credits; CloudWatch accepts an X-Ray API key as a bearer header.

Rules that keep it safe.

  • Off by default. With the switch off, OTEL_EXPORTER_OTLP_* variables set for another tool send nothing.
  • The endpoint must be https:// unless the host is loopback, and carry no user:password or query.
  • A header value written in config is refused, not sent. Headers are read from the vault or an environment variable, and environment headers go only to an environment endpoint.
  • No log line from this path quotes a header value; header checks reject anything requests would refuse, and request errors are replaced with fixed text.
  • The export session follows no redirect and never reads ~/.netrc, so the collector receives only the headers you configured.
  • kirocrew.local.* keys never reach the wire.
  • LTS builds ignore every local traces endpoint setting.

Known limitations.

  • A collector that echoes credentials back can cause them to appear in local logs. urllib3 logs a malformed response header line as it reads it, before Kiro Crew sees the response. Closing that needs a filter on urllib3's process-wide loggers, which other HTTP clients share, so it is documented in the spec and the traces guide instead.
  • The process-wide OTEL_EXPORTER_OTLP_*CERTIFICATE, CLIENT_KEY and COMPRESSION variables still apply to a configured endpoint, because the SDK exporter reads them for any argument left empty. No secret is sent, but a client certificate set for another collector is presented to this one.
  • A rotated vault key is read when tracing is switched off and on, or on restart.

A new guide, docs/guides/telemetry-otlp-traces.md, walks through Langfuse, CloudWatch (API key and collector) and a credits query.

Where.

  • src/kiro_crew/tracing/otlp.py (new): where spans may go, header reading, the export session, the wire filter for local keys.
  • src/kiro_crew/tracing/provider.py: adds the OTLP batch processor on the first collector, swaps it on a config write, pins batch limits; tracing/__init__.py names the module.
  • src/kiro_crew/metrics/provider.py and metrics/otlp_governance.py: the policy check and per-export gate, reused for spans.
  • src/kiro_crew/config/ (sections_telemetry.py, loader_builders.py, loader.py) and config-baseline.json: the three settings; src/kiro_crew/edition_locks.py: their LTS lock.

Backwards compatibility

Compatible: every new setting is off or empty by default, and with tracing off nothing changes. gate_exporter gains an optional refused argument; existing callers pass nothing and keep the metric result.

Removes nothing: the change only adds settings, a module and a guide; the metrics /v1/traces warning still fires and now points at the traces setting.

Tests

test/tracing/test_tracing_otlp.py, against an in-process receiver bound to 127.0.0.1:

  • a traced turn reaches the receiver with its vault header, the Langfuse keys, OTEL_SERVICE_NAME, and no kirocrew.local.* key; the local file keeps them;
  • a host the policy denies receives nothing, and a policy installed later stops the next export;
  • an environment endpoint sends nothing while the switch is off;
  • no header value appears in any log record on 200, 400 or 401, or when a library error or the collector's reason phrase echoes it;
  • 307 and 308 are not followed, so the redirect target receives nothing;
  • a .netrc entry for the collector's host is never sent, with or without a vault header;
  • a rotated vault key is sent after tracing is switched off and on;
  • a credential pasted as a variable name, and a policy refusal, each log one warning that quotes no value;
  • each header check pins its character set and bounds: empty, too long, CR/LF, NUL, non-ASCII and a trailing newline are refused;
  • the LTS lock ignores config and environment endpoints;
  • the sandbox probe stays single-threaded with export on (the Sandbox probe cannot answer in a process that restarts a thread at fork (telemetry-enabled gateway) #4219 class);
  • both batch processors use the pinned limits whatever OTEL_BSP_* says, and the exit flush sends queued spans.

Determinism: the file passed 5 repeats and a reversed order; it binds only an ephemeral loopback port and waits on no wall-clock delay.

Manual verification

Still required, needs AWS: one real Claude turn against CloudWatch Transaction Search, confirming that model and token fields fill (RFC phase 2 exit criterion), and the guide's Logs Insights credits query.

Related Issues

Part of #1643

RFC: #31433. Builds on #31460 (phase 1a) and #35077 (phase 1b). Whole-RFC acceptance: #35443.

Deferred: the live CloudWatch check above (needs AWS), phase 2b native SigV4, phase 7 Settings UI.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@kyleseaman
kyleseaman requested review from a team and KiroCrewCaptain as code owners November 12, 2026 08:46
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 12, 2026
@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 915477c6dd3a3ad672d76d12e811822962f946da touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 915477c6dd3a3ad672d76d12e811822962f946da — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks. Span export goes through the same otlp_hosts policy, audit row, per-export gate and LTS lock that metrics already use. It is off by default and takes nothing away.

Notes

  • metrics/provider.py was already about 1,960 lines on the base, and this PR adds two trace helpers to it. That is the wrong home under the 1,600-line rule. Putting them next to gate_exporter in metrics/otlp_governance.py would keep the shared policy seam in one place without growing the large file.
  • service.name now comes from OTEL_SERVICE_NAME, and it changes the local JSONL resource as well as what goes over the wire. If another tool sets that variable, it renames traces already on the host. Mention this in the traces guide, or read the variable only when building the OTLP exporter.

[DESIGN-REVIEWED] 915477c

Description read

[DESCRIPTION-READ] 6f31ebaccc88d3f9ed000ff9275fa5a863202e8707c5a3b4816f78b4f3402edd

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of 915477c6dd3a3ad672d76d12e811822962f946da — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Every item traces to the phase 2 row of rfc-otel-tracing.md, which is in-progress on base. The only finding is a duplicated loopback helper.

Notes

  • Item 5 — duplicate of relay_client/protocol.py:623 is_loopback_host: tracing/otlp.py:144 _is_loopback adds an 11th private loopback check (grepped def _?is_loopback\w*\(, 10 existing sites). Subtraction: delete _is_loopback and call relay_client.protocol.is_loopback_host.

What this change ships

Inventory (10 items) — 8 justified

Intent: let an operator see Kiro Crew's per-turn traces in Langfuse, CloudWatch or any OTLP collector. ADDITION.

  1. With traces on and traces_otlp_endpoint set, each turn's spans reach that collector — justified
  2. Auth headers come from Settings → Secrets entries named in traces_otlp_headers_secret — justified
  3. Headers can also come from environment variables via traces_otlp_headers_env — justified
  4. OTEL_EXPORTER_OTLP_* endpoint and header variables are used only while the switch is on — justified
  5. Endpoint must be https unless loopback, with no user:password or query; a literal header value in config is refused — duplicate of relay_client/protocol.py:is_loopback_host
  6. The otlp_hosts policy and SEL audit are checked when the exporter is built and on every export — justified
  7. LTS builds ignore every traces endpoint and header setting — justified
  8. No redirects, no .netrc, error text replaced with fixed text, kirocrew.local.* keys stripped before sending — justified
  9. The trace resource's service.name comes from OTEL_SERVICE_NAME when it is set — justified
  10. Both batch processors use pinned limits, and the local sink's export timeout drops from the SDK's 30s to 10s — rides along

[FIRST-PRINCIPLES-REVIEWED] 915477c

Description read

[DESCRIPTION-READ] 84a2365b88f52b6a9b462e8a8bc906cc4c49ea10a59ea5e3c32bfdff2d4801cc

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 915477c6dd3a3ad672d76d12e811822962f946da — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 915477c

Verdict parsed from the review's SHA-scoped output markers for commit 915477c6dd3a3ad672d76d12e811822962f946da.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 915477c6dd3a3ad672d76d12e811822962f946da false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ human override accepted

Human judgment by @kyleseaman overrides the GPT 6.1 finding for 915477c6dd3a3ad672d76d12e811822962f946da; the recorded reason is authoritative for this commit.

[GPT-OVERRIDE] 915477c

This comment is updated in place on each push.

The model was not re-run because an authorized human decision supersedes it.

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 915477c6dd3a3ad672d76d12e811822962f946da false-positive: <reason, 20 to 500 characters>

@kyleseaman
kyleseaman force-pushed the crew/devmate/otel-pr2 branch from ed0f853 to c61c97e Compare November 12, 2026 09:02
@kyleseaman

Copy link
Copy Markdown
Contributor Author

span=178890badd65

  • Collector responses can expose vault-backed authentication in logs

rebutted: not a defect in this change. The trigger needs the collector the operator configured, which already holds the credential, to echo it back as a malformed response header; the value then lands only in this host's own log, which the operator owns. No new party can read it.
The line is written by urllib3 (connection.py, HeaderParsingError with exc_info=True) inside requests, before any Kiro Crew code sees the response. Closing it needs a filter on urllib3's process-wide loggers, which every other HTTP client in the gateway shares, and which would itself be a redaction sink under Sensitive Change Review.
Everything this PR owns is closed: header values are refused before requests sees one it would reject, request errors are re-raised with fixed text, reason phrases are replaced, redirects are refused and .netrc is never read (src/kiro_crew/tracing/otlp.py).
Recorded as a known limitation in the PR body, docs/system-specs/modules/metrics.md (Traces, Export) and docs/guides/telemetry-otlp-traces.md, per the project's stopping rule for this finding class. Any override is left to a maintainer.

@KiroCrewCaptain

KiroCrewCaptain commented Nov 12, 2026 •

Copy link
Copy Markdown
Collaborator

Code change approver recommendation: A person should decide

Why: the decider thinks the change is probably fine but estimates a 15% chance that this change introduces a defect, above the 12% limit for approving on its own.

Adds opt-in OTLP span export through the trust path metrics already uses. Checked in the code:

  • Off by default; an environment endpoint sends nothing while the switch is off.
  • LTS builds ignore every config and environment traces endpoint, and the three new keys are lock-mapped.
  • Endpoint and header rules hold: https unless loopback, no userinfo/query/fragment, literal header values refused, warnings never quote a value.
  • A config endpoint gets only its own headers (_OwnHeadersOnly).
  • The otlp_hosts policy is checked and SEL-audited at build and on every export; the new refused argument leaves metric callers unchanged.
  • The session follows no redirect, skips netrc and re-raises request errors with fixed text.
  • kirocrew.local.* keys are stripped from spans and links.
  • Config writes swap, clear, keep or retry the exporter correctly, and shutdown flushes the local sink first.
  • config.json is sealed against sandboxed agents, so naming vault entries in config does not widen the trust boundary.
    Tests cover each claimed behaviour. Remaining items are process or documentation notes: merge conflict, pending CI, the manual AWS check, file size and a duplicated helper.

An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit 915477c6dd · 1957 changed lines. If this recommendation is wrong, reply here: replies are how the approver is evaluated.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention merge conflict Branch has merge conflicts with its base — author must resolve before merge and removed readiness: checking Automated validation is still running labels Nov 12, 2026
@kyleseaman

Copy link
Copy Markdown
Contributor Author

/ai-review override gpt c61c97e false-positive: documented limitation (collector echoing credentials into local urllib3 WARNING logs); the only fix is a process-wide log filter, which would be a redaction sink requiring Sensitive Change Review. Tracked as a known limitation in the spec and guide.

@github-actions

Copy link
Copy Markdown
Contributor

Human judgment recorded

@kyleseaman recorded a false-positive release (class false-positive) for the gpt AI finding on c61c97edbce557130f20815ec64177db5cb01a92.

documented limitation (collector echoing credentials into local urllib3 WARNING logs); the only fix is a process-wide log filter, which would be a redaction sink requiring Sensitive Change Review. Tracked as a known limitation in the spec and guide.

This decision applies only to this commit. A new push requires a new judgment.

@kyleseaman
kyleseaman force-pushed the crew/devmate/otel-pr2 branch from c61c97e to 915477c Compare November 12, 2026 15:40
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Nov 12, 2026
@kyleseaman

Copy link
Copy Markdown
Contributor Author

Rebased: main moved in docs/system-specs/modules/metrics.md since this head went green (386875e -> 5e7b587); CI re-running.

The rebase resolved one docs-only conflict in the Traces paragraph (main #35651 reworded the watch_config() sentence). It keeps main's wording and adds this PR's traces_otlp_* inputs. Code is unchanged: every non-docs file adds and removes the same lines as c61c97e. New head: 915477c.

@github-actions github-actions Bot removed the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Nov 12, 2026
@kyleseaman

Copy link
Copy Markdown
Contributor Author

/ai-review override gpt 915477c false-positive: documented limitation (collector echoing credentials into local urllib3 WARNING logs); the only fix is a process-wide log filter, which would be a redaction sink requiring Sensitive Change Review. Tracked as a known limitation in the spec and guide.

@github-actions

Copy link
Copy Markdown
Contributor

Human judgment recorded

@kyleseaman recorded a false-positive release (class false-positive) for the gpt AI finding on 915477c6dd3a3ad672d76d12e811822962f946da.

documented limitation (collector echoing credentials into local urllib3 WARNING logs); the only fix is a process-wide log filter, which would be a redaction sink requiring Sensitive Change Review. Tracked as a known limitation in the spec and guide.

This decision applies only to this commit. A new push requires a new judgment.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Nov 12, 2026
@KiroCrewCaptain

Copy link
Copy Markdown
Collaborator

This adds a new opt-in path that sends trace data off the machine, and #1643 is a T3 item that waits on a design decision, so it needs a design sign-off from a maintainer before it can merge. A maintainer will post the decision here.

The review found no blocking defect on this head. It checked these points in the diff:

  • Span export stays off unless traces_enabled is set, and it is locked on LTS.
  • Endpoints must use https unless they are loopback, and userinfo, query and fragment are refused.
  • A config endpoint never receives the OTEL_*HEADERS variables, and config holds only vault or env names, never header values.
  • The otlp_hosts policy is checked when the exporter is built and again on every export.
  • Redirects and ~/.netrc are refused.
  • kirocrew.local.* keys are stripped before export.

The only design acceptance on record is the RFC author's own comment on #1643, and #35443 is not merged yet.

Security: the PR author overrode the one security lane finding as a false positive. That finding is that a collector which echoes a vault-backed Authorization value in a malformed response header can get that value logged by urllib3. The rebuttal reads as sound: the trigger needs the operator's own collector, and the guide documents the risk. But it is a real limitation that has been accepted, so a maintainer should record that release under the class that fits it.

Done well: the PR parses OTEL_EXPORTER_OTLP_HEADERS itself, because the SDK's parser logs malformed entries verbatim. The remote exporter also sits behind its own batch processor, so a slow collector never delays a turn or the local sink.

Optional:

  • The wire-level tests in test/tracing/test_tracing_otlp.py skip under requires_otlp, and no CI job installs the OTLP exporter. So the netrc, redirect, header-echo and local-key-strip checks never run in CI. You could install the exporter in one backend lane, or add tests that need only requests.
  • src/kiro_crew/metrics/provider.py grows to 1985 lines, 15 under the 2000-line cap. Moving trace_destination_permitted and gate_trace_exporter into metrics/otlp_governance.py or tracing/otlp.py would keep the file at its current size.
  • If the destination is refused when the exporter is built, export is only tried again after a later trace-setting write or a restart. This happens when the policy cannot be evaluated, or when the secret is created after the config write. One sentence in the guide, or a "build refused" flag in the inputs fingerprint, would cover it.

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Export phase direction signed by maintainer (bolichen97): opt-in, default off; whole-RFC acceptance still tracked in #35443.

I read the full diff: config, edition_locks, otlp_governance, metrics/provider, tracing/otlp.py, tracing/provider.py and the tests.

  • Nothing leaves without a collector. resolve returns None unless traces_enabled is truly on and a config or OTEL_EXPORTER_OTLP_* endpoint is named. Without one, no OTLP processor or thread is added, and the test counts only the local sink's thread. Switching off clears the exporter, so spans a still-open turn ends go nowhere. An LTS build ignores all three endpoint sources.
  • What leaves. RemoteSpans strips every kirocrew.local.* key from spans and links before export. The wire test shows the raw session key only in the local sink.
  • Headers. Values are read only from Settings -> Secrets entries or named variables. A literal value in config is refused. A config endpoint never gets the process-wide OTel header variables, and no warning quotes a value or a vault reference.
  • Transport. No redirect is followed, .netrc is never read, and requests errors and reason phrases are rewritten so a value cannot reach a log.
  • Endpoint rules. https is required unless the host is loopback, with no user:password or query. The host is held to otlp_hosts at build and again on every export, with SEL audit.
  • No regex in product code. The character checks are sets with length bounds.

The checks the feature adds apply only to its own new inputs; no path main has is tightened. All checks green on this head. LGTM.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

readiness: passed Eligible automated validation passed for the current revision

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants