Repository navigation
ci: install the otlp extra in backend test lanes so wire-level OTLP tests run - #36721
Conversation
|
Local review before the first push (
Low:
Local gates: |
UX Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing blocks: the PR installs one pinned extra in the three backend test lanes and adds a test-only reset for the cached tracing switch, which is the cause of the measured order-dependent failure. [DESIGN-REVIEWED] b07bc06 Description read[DESCRIPTION-READ] 101a276c35850aa02e456378eef2da98563915e45caa79c794cf1c2e4d9afc89 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Nothing blocks: the extra closes the CI gap reported in #36677. The reset fixture fixes a test-isolation failure that the PR reproduced, and it removes the stale cache at its source. What this change shipsInventory (4 items) — 4 justifiedIntent: make CI actually run the wire-level OTLP export security tests (no redirect, no
[FIRST-PRINCIPLES-REVIEWED] b07bc06 Description read[DESCRIPTION-READ] 33d5e5f7401cc32da7dde38f84392e68c8e452a2debe333ecd1aa505fe558a97 That sha256 names the PR description this verdict read. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
a78be67 to
f40d6df
Compare
|
Windows red on a78be67:
Evidence: run Not rebased: |
Code change approver recommendation: ApproveWhy: the decider estimates a 6% chance that this change introduces a defect, within the 12% limit for approving on its own, and every check passes. Adds the pinned An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit |
|
Coverage Gate red on f40d6df is main's, not this PR's. It fails on
|
f40d6df to
b07bc06
Compare
|
Rebased: main moved in .github/workflows/codex-review.yml (same-dir) and test/metrics/test_cost_breakdown.py (same-dir) since this head went green (6a35758 -> bd08975); CI re-running. This picks up #36804 for the main-wide Coverage Gate red ( Head f40d6df -> b07bc06. Same |
bolichen97
left a comment
There was a problem hiding this comment.
Approve. Read the whole diff (ci.yml, platform-tests.yml, ci-and-reviews.md, test/conftest.py, test/conftest_resets.py, two test docstrings).
- Test-only:
otlpis added only to theuv pip installlines of the backend test lanes (Linux, Windows, platform tests).setup.cfgis not touched, so the shipped default install is the same as on main;otlpstays an optional extra (setup.cfg:172). - The new autouse fixture
_forget_tracing_switchcallskiro_crew.tracing.provider.reset_for_testing()(on main,provider.py:409) at setup, and only when the module is already imported. Resetting at setup, not teardown, keeps the leaked-exporter check working. - No product code change, so there is no new refusal and no regex in product code.
Checks: 104 ok, 0 red, 0 pending on this head. I did not open the shard logs to count how many requires_otlp tests ran.
#37005 adds one kirocrew.step span per derived model call to a traced turn. The two wire-level OTLP tests still expected the four spans from before, so they fail wherever the otlp extra is installed, which CI does since #36721. Expect the step span and a count of five. Closes #37169 (cherry picked from commit 6de8528)
…irodotdev#36721) (cherry picked from commit 93a063c, reset hunks only)
…irodotdev#36721) (cherry picked from commit 93a063c, reset hunks only)
Problem / Motivation
Goal: CI runs the wire-level OTLP tests, so the export security checks (no redirect, no
.netrc, no header echo) are tested on every PR instead of only on a developer machine.No backend test lane installs the
otlpextra (setup.cfg:otlp = opentelemetry-exporter-otlp-proto-http==1.44.0). Every test guarded byrequires_otlp(or the same import check) skips in CI: 44 tests acrosstest/tracing/test_tracing_otlp.py,test/metrics/test_otlp_wire_e2e.py,test/metrics/test_otlp_endpoint_list.pyand all oftest/metrics/test_otlp_no_redirect.py. KiroCrewCaptain flagged the gap on #35792.Why it matters
The OTLP exporters send metrics and spans off the machine. The tests that prove a real exporter follows no redirect, ignores
~/.netrcand never echoes a header value have never run in CI, so a dependency bump or a refactor could break them and still go green.Not a goal
What changed (motivation → approach → change)
The three backend test lanes now install the
otlpextra, so the 44 skipped OTLP tests run on Linux, Windows and macOS.backend-test(Linux, 8 shards)-e ".[voice,desktop,otlp]"backend-test-windows-e ".[voice,desktop,otlp]"platform-tests.yml(macOS nightly and on-demand)-e ".[voice,desktop,otlp]"Running them exposed a test-isolation bug. The tracing switch (
kiro_crew.tracing.provider) caches its first answer for the whole worker. A test that boots the gateway services (test_slack_gateway.py'stest_init_services_creates_all) caches "off", so a later wire test on the same worker exported nothing and failed withno OTLP request reached the receiver. That happened on Windows shard 2. A new autouse fixture,_forget_tracing_switchintest/conftest_resets.py, resets the switch before every test. No product code changes.The
backend-testrow indocs/ci/ci-and-reviews.mdsays which extras these lanes install. Two test docstrings that said CI lacks the extra now say a default dev install lacks it.Coverage stays safe both ways.
test_otlp_session.pystubs the SDK module, so it still covers every branch ofmetrics/otlp_session.pywith the extra present. Measured ontest/tracing+test/metrics, the lines missed with the extra are a subset of those missed without it (for exampletracing/otlp.py84% → 92%,metrics/provider.py90% → 91%). No metrics or tracing file is in.github/coverage-baselines/backend.txt, so the per-file ratchet cannot trip.Backwards compatibility
Compatible: CI installs one more pinned distribution; no product code, config key or default changes.
Removes nothing: the diff adds an extra to three install lines, adds one test-only reset fixture and rewords comments.
Tests
No new tests. The 44 existing
requires_otlptests now run in CI.Isolation fix, failing first:
test_init_services_creates_allfollowed bytest_a_traced_turn_reaches_a_loopback_receiver_with_its_vault_header(-n0, that order) fails on the parent withno OTLP request reached the receiverand passes with_forget_tracing_switch. The order is the one Windows shard 2's workergw0ran; a probe that stops at the first test leaving the switch resolved foundtest_init_services_creates_allin that worker's earlier files. With the fixture: 95 tracing-related test files plus that worker's culprit chunk, 3229 passed;test/tracing+test/metrics+ the gateway boot tests, 2209 passed.Determinism of the newly running tests, with the extra installed: the 212 tests in the five OTLP files passed in three seeded shuffled serial orders (seeds 11, 22, 33) and once with
-n 4.Local runs with the extra:
test/tracing+test/metrics2097 passed, 0 skipped (without it: 2053 passed, 44 skipped). The 18 other test files that mention OTLP: 810 passed. Scoped backend tests: 4065 passed. The flake-pattern ratchet and semgrep (diff-scoped) pass.Manual verification
Linux only. The Windows and macOS lanes are verified by this PR's own CI: editing
platform-tests.ymltriggersmacOS Tests (on demand).Related Issues
Closes #36677
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)