Skip to content

ci: install the otlp extra in backend test lanes so wire-level OTLP tests run - #36721

Merged
bolichen97 merged 1 commit into
mainfrom
crew/devmate/ci-otlp-extra
Nov 14, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
crew/devmate/ci-otlp-extra

Conversation

@kyleseaman

@kyleseaman kyleseaman commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: CI runs the wire-level OTLP tests, so the export security checks (no redirect, no .netrc, no header echo) are tested on every PR instead of only on a developer machine.

No backend test lane installs the otlp extra (setup.cfg: otlp = opentelemetry-exporter-otlp-proto-http==1.44.0). Every test guarded by requires_otlp (or the same import check) skips in CI: 44 tests across test/tracing/test_tracing_otlp.py, test/metrics/test_otlp_wire_e2e.py, test/metrics/test_otlp_endpoint_list.py and all of test/metrics/test_otlp_no_redirect.py. KiroCrewCaptain flagged the gap on #35792.

Why it matters

The OTLP exporters send metrics and spans off the machine. The tests that prove a real exporter follows no redirect, ignores ~/.netrc and never echoes a header value have never run in CI, so a dependency bump or a refactor could break them and still go green.

Not a goal

  • Installing the extra in every CI job. Lint, E2E, IPv6, sandbox, container and new-test-stability lanes keep their current installs.
  • Changing any OTLP code or test logic.
  • Making the extra a default dependency.

What changed (motivation → approach → change)

The three backend test lanes now install the otlp extra, so the 44 skipped OTLP tests run on Linux, Windows and macOS.

Lane Install line
backend-test (Linux, 8 shards) -e ".[voice,desktop,otlp]"
backend-test-windows -e ".[voice,desktop,otlp]"
platform-tests.yml (macOS nightly and on-demand) -e ".[voice,desktop,otlp]"

Running them exposed a test-isolation bug. The tracing switch (kiro_crew.tracing.provider) caches its first answer for the whole worker. A test that boots the gateway services (test_slack_gateway.py's test_init_services_creates_all) caches "off", so a later wire test on the same worker exported nothing and failed with no OTLP request reached the receiver. That happened on Windows shard 2. A new autouse fixture, _forget_tracing_switch in test/conftest_resets.py, resets the switch before every test. No product code changes.

The backend-test row in docs/ci/ci-and-reviews.md says which extras these lanes install. Two test docstrings that said CI lacks the extra now say a default dev install lacks it.

Coverage stays safe both ways. test_otlp_session.py stubs the SDK module, so it still covers every branch of metrics/otlp_session.py with the extra present. Measured on test/tracing + test/metrics, the lines missed with the extra are a subset of those missed without it (for example tracing/otlp.py 84% → 92%, metrics/provider.py 90% → 91%). No metrics or tracing file is in .github/coverage-baselines/backend.txt, so the per-file ratchet cannot trip.

Backwards compatibility

Compatible: CI installs one more pinned distribution; no product code, config key or default changes.

Removes nothing: the diff adds an extra to three install lines, adds one test-only reset fixture and rewords comments.

Tests

No new tests. The 44 existing requires_otlp tests now run in CI.

Isolation fix, failing first: test_init_services_creates_all followed by test_a_traced_turn_reaches_a_loopback_receiver_with_its_vault_header (-n0, that order) fails on the parent with no OTLP request reached the receiver and passes with _forget_tracing_switch. The order is the one Windows shard 2's worker gw0 ran; a probe that stops at the first test leaving the switch resolved found test_init_services_creates_all in that worker's earlier files. With the fixture: 95 tracing-related test files plus that worker's culprit chunk, 3229 passed; test/tracing + test/metrics + the gateway boot tests, 2209 passed.

Determinism of the newly running tests, with the extra installed: the 212 tests in the five OTLP files passed in three seeded shuffled serial orders (seeds 11, 22, 33) and once with -n 4.

Local runs with the extra: test/tracing + test/metrics 2097 passed, 0 skipped (without it: 2053 passed, 44 skipped). The 18 other test files that mention OTLP: 810 passed. Scoped backend tests: 4065 passed. The flake-pattern ratchet and semgrep (diff-scoped) pass.

Manual verification

Linux only. The Windows and macOS lanes are verified by this PR's own CI: editing platform-tests.yml triggers macOS Tests (on demand).

Related Issues

Closes #36677

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@kyleseaman
kyleseaman requested review from a team and KiroCrewCaptain as code owners November 13, 2026 16:11
@kyleseaman

Copy link
Copy Markdown
Contributor Author

Local review before the first push (kirocrew-prepare-pr Phase 2), on reviewed head 958f4a479bd4. It was rebased without conflict onto 6a35758 as a78be67, which has the same git patch-id. Main did not touch any file in this diff in that window.

  • GPT (gpt-6.1-sol, codex-review.yml contract): no findings.
  • Opus (claude-opus-5.5, claude-review.yml contract): no Critical/High/Medium findings. One Low finding is answered below.

Low: test-durations.yml:59 still installs .[voice,desktop], so the recorded shard timings count the OTLP tests as about 0 s skips.

Rebutted: disproportional. The cost is a small shard imbalance and nothing fails. All five OTLP test files (212 tests, the 44 newly running included) take about 21 s serial, under 3 s per shard spread over 8 shards. The durations workflow builds timing data and does not run backend tests, so it sits outside this PR's goal of making the tests run. Its timings pick up the extra if that lane ever installs it.

Local gates: pre_push_gate.py 21 ran, 0 failed (Gate proof: tree 4f40d2fbdb3f; ran 21; skipped none). Scoped backend tests: 4065 passed. Flake-pattern ratchet and diff-scoped semgrep: pass. The frontend gates were not run because the diff touches no frontend file; CI runs them.

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision b07bc0643da15f63cad50f3e98eaea307aac9e0c touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of b07bc0643da15f63cad50f3e98eaea307aac9e0c — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: the PR installs one pinned extra in the three backend test lanes and adds a test-only reset for the cached tracing switch, which is the cause of the measured order-dependent failure.

[DESIGN-REVIEWED] b07bc06

Description read

[DESCRIPTION-READ] 101a276c35850aa02e456378eef2da98563915e45caa79c794cf1c2e4d9afc89

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of b07bc0643da15f63cad50f3e98eaea307aac9e0c — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Nothing blocks: the extra closes the CI gap reported in #36677. The reset fixture fixes a test-isolation failure that the PR reproduced, and it removes the stale cache at its source.

What this change ships

Inventory (4 items) — 4 justified

Intent: make CI actually run the wire-level OTLP export security tests (no redirect, no .netrc, no header echo), not skip them. FIX.

  1. The Linux, Windows and macOS backend test lanes install the otlp extra, so the 44 requires_otlp tests run instead of skipping — justified
  2. Each test now starts with the tracing switch unread (_forget_tracing_switch), so a gateway-boot test can no longer leave "off" cached for later tests — justified
  3. The backend-test row in the CI doc names the extras these lanes install — justified
  4. Two test docstrings now say a default dev install lacks the extra, not CI — justified

[FIRST-PRINCIPLES-REVIEWED] b07bc06

Description read

[DESCRIPTION-READ] 33d5e5f7401cc32da7dde38f84392e68c8e452a2debe333ecd1aa505fe558a97

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed b07bc0643da15f63cad50f3e98eaea307aac9e0c — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] b07bc06

Verdict parsed from the review's SHA-scoped output markers for commit b07bc0643da15f63cad50f3e98eaea307aac9e0c.

False positive or not applicable? A repository writer can comment:
/ai-review override fable b07bc0643da15f63cad50f3e98eaea307aac9e0c false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of b07bc0643da15f63cad50f3e98eaea307aac9e0c and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] b07bc06

False positive or not applicable? A repository writer can comment:
/ai-review override gpt b07bc0643da15f63cad50f3e98eaea307aac9e0c false-positive: <reason, 20 to 500 characters>

@github-actions github-actions Bot added the readiness: action required A blocking check or review needs attention label Nov 13, 2026
@kyleseaman
kyleseaman force-pushed the crew/devmate/ci-otlp-extra branch from a78be67 to f40d6df Compare November 13, 2026 18:48
@kyleseaman

Copy link
Copy Markdown
Contributor Author

Windows red on a78be67: Backend Tests (Windows) (2), test_a_traced_turn_reaches_a_loopback_receiver_with_its_vault_header

fixed in f40d6df: a test-isolation bug, not an OTLP or Windows-encoding defect. test_slack_gateway.py::TestInitServices::test_init_services_creates_all boots the gateway services, and service_boot calls tracing.provider.watch_config(). That resolves the tracing switch to "off" and caches it for the worker (_resolved=True, _state=None), so a later wire test on worker gw0 saw tracing off and exported nothing. The UnicodeDecodeError (byte 0x97) in the same log comes from a reader thread in test_embedding_rebuild_process.py and is unrelated. A new autouse _forget_tracing_switch fixture in test/conftest_resets.py resets the switch at setup. No product code changes.

Evidence: run -n0 in that order, the pair fails on the parent with no OTLP request reached the receiver and passes with the fixture; reproduced on Linux. I found the culprit by replaying gw0's files from the shard's progress artifact with a probe that stops at the first test leaving the switch resolved. With the fixture: 95 tracing-related files plus that chunk, 3229 passed; test/tracing + test/metrics + gateway boot tests, 2209 passed. Local GPT (gpt-6.1-sol) and Opus (claude-opus-5.5) reviews of the interdiff a78be67..f40d6df: no findings. Pre-push gates: 21 ran, 0 failed. Semgrep and the flake ratchet pass.

Not rebased: git merge-tree --write-tree refs/remotes/origin/main HEAD exits 0, so main (now 4c43c24) does not conflict with this diff.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Nov 13, 2026
@KiroCrewCaptain

KiroCrewCaptain commented Nov 13, 2026 •

Copy link
Copy Markdown
Collaborator

Code change approver recommendation: Approve

Why: the decider estimates a 6% chance that this change introduces a defect, within the 12% limit for approving on its own, and every check passes.

Adds the pinned otlp extra to exactly the three backend test lanes (Linux, Windows, macOS); every other install line is unchanged, as the PR's non-goals say. Adds an autouse test fixture that resets the cached tracing switch at setup through tracing.provider.reset_for_testing(). That call shuts down any built tracer and clears the cache. It does not conflict with the leak check's before/after thread comparison, which runs at teardown. No tracing test relies on module- or session-scoped tracer state. All backend shards, including Windows and macOS, plus the coverage and dependency-license gates, are green on this head. No product code changes.

An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit b07bc0643d · 40 changed lines. If this recommendation is wrong, reply here: replies are how the approver is evaluated.

@kyleseaman

Copy link
Copy Markdown
Contributor Author

Coverage Gate red on f40d6df is main's, not this PR's. It fails on src/kiro_crew/agent_ipc_client.py 75.7% (87/115), a file this diff does not touch. The same file fails main's Coverage Gate at 4c43c24 (run 43484262760, job 132110344206). #36804 adds the missing coverage. Once it merges this PR rebases onto it and re-runs. No push or override for this red until then.

Analyze (python) and Analyze (javascript-typescript) failed on API rate limit exceeded for installation; the failed jobs of run 43486182011 were rerun once.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 13, 2026
@kyleseaman
kyleseaman force-pushed the crew/devmate/ci-otlp-extra branch from f40d6df to b07bc06 Compare November 13, 2026 20:42
@kyleseaman

Copy link
Copy Markdown
Contributor Author

Rebased: main moved in .github/workflows/codex-review.yml (same-dir) and test/metrics/test_cost_breakdown.py (same-dir) since this head went green (6a35758 -> bd08975); CI re-running. This picks up #36804 for the main-wide Coverage Gate red (src/kiro_crew/agent_ipc_client.py at 75.7%). CI pins each run to its merge commit, so a rerun could not have picked the fix up.

Head f40d6df -> b07bc06. Same git patch-id, so the change is the one reviewed. git merge-tree against main exited 0 (no conflict). Push guard: single commit on base. Related tests (test/tracing, test/metrics, test/test_slack_gateway.py, test/test_shared_crew_admission_entry.py): 2182 passed, and the Windows-shard ordering pair passes.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Nov 13, 2026

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve. Read the whole diff (ci.yml, platform-tests.yml, ci-and-reviews.md, test/conftest.py, test/conftest_resets.py, two test docstrings).

  • Test-only: otlp is added only to the uv pip install lines of the backend test lanes (Linux, Windows, platform tests). setup.cfg is not touched, so the shipped default install is the same as on main; otlp stays an optional extra (setup.cfg:172).
  • The new autouse fixture _forget_tracing_switch calls kiro_crew.tracing.provider.reset_for_testing() (on main, provider.py:409) at setup, and only when the module is already imported. Resetting at setup, not teardown, keeps the leaked-exporter check working.
  • No product code change, so there is no new refusal and no regex in product code.

Checks: 104 ok, 0 red, 0 pending on this head. I did not open the shard logs to count how many requires_otlp tests ran.

@bolichen97
bolichen97 merged commit 93a063c into main Nov 14, 2026
113 of 120 checks passed
@bolichen97
bolichen97 deleted the crew/devmate/ci-otlp-extra branch November 14, 2026 07:03
buluoray added a commit that referenced this pull request Nov 14, 2026
#37005 adds one kirocrew.step span per derived model call to a traced
turn. The two wire-level OTLP tests still expected the four spans from
before, so they fail wherever the otlp extra is installed, which CI does
since #36721. Expect the step span and a count of five.

Closes #37169
chenmingwei23 pushed a commit that referenced this pull request Nov 15, 2026
#37005 adds one kirocrew.step span per derived model call to a traced
turn. The two wire-level OTLP tests still expected the four spans from
before, so they fail wherever the otlp extra is installed, which CI does
since #36721. Expect the step span and a count of five.

Closes #37169

(cherry picked from commit 6de8528)
jeeshofone pushed a commit to jeeshofone/KiroCrew that referenced this pull request Nov 16, 2026
jeeshofone pushed a commit to jeeshofone/KiroCrew that referenced this pull request Nov 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

readiness: passed Eligible automated validation passed for the current revision

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CI skips the requires_otlp wire tests: no backend lane installs the otlp extra

3 participants