Skip to content

test: expect the kirocrew.step span in the otlp wire tests - #37171

Merged
buluoray merged 1 commit into
mainfrom
crew/slack/fix-otlp-step-span
Nov 14, 2026
Merged

buluoray merged 1 commit into
mainfrom
crew/slack/fix-otlp-step-span

Conversation

@buluoray

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Main's backend test shard 2 goes green again by making the two wire-level OTLP tests expect the kirocrew.step span that #37005 added.

Every pull request that contains #37005 fails Backend Tests (3.12, 2) and Backend Tests (Windows) (2) on test/tracing/test_tracing_otlp.py: one test sees an extra kirocrew.step span, the other counts 5 spans where it expects 4.

Why it matters

Main is red for every open PR rebased after 08:30Z. Authors either rerun a test that cannot pass or read the red as their own bug.

Not a goal

What changed (motivation → approach → change)

#37005 adds one kirocrew.step span per derived model call. The two OTLP wire tests still listed the four spans from before. They are marked requires_otlp, so they skip on a machine without the otlp extra, which is why the change passed locally; CI installs the extra since #36721 and runs them. The fix adds kirocrew.step to the expected span set in test_a_traced_turn_reaches_a_loopback_receiver_with_its_vault_header and makes the count 5 in test_the_exit_flush_sends_spans_still_queued.

Backwards compatibility

Compatible: only two test expectations change.

Removes nothing: no product code changes.

Tests

  • test/tracing/test_tracing_otlp.py: the two tests now expect the step span.

Proof: not provable -- test-only change

Run with opentelemetry-exporter-otlp-proto-http==1.44.0 installed (the otlp extra):

# on main 23b00bfa78a, unchanged
FAILED test/tracing/test_tracing_otlp.py::test_a_traced_turn_reaches_a_loopback_receiver_with_its_vault_header - AssertionError: assert {'chat', 'inv...'user.prompt'} == {'chat', 'inv...'u...
FAILED test/tracing/test_tracing_otlp.py::test_the_exit_flush_sends_spans_still_queued - AssertionError: assert 5 == 4
2 failed, 91 passed

# with this change
test/tracing/: 310 passed
test/tracing/test_tracing_otlp.py, three runs in a row: 93 passed, 93 passed, 93 passed

Determinism: this change touches only the expected values; the three repeat runs above passed.

Manual verification

N/A — the change is two expected values in tests, shown red on main and green here with the extra installed.

Related Issues

Closes #37169

Follows #37005 (the step span) and #36721 (the otlp extra in CI).

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

#37005 adds one kirocrew.step span per derived model call to a traced
turn. The two wire-level OTLP tests still expected the four spans from
before, so they fail wherever the otlp extra is installed, which CI does
since #36721. Expect the step span and a count of five.

Closes #37169
@buluoray
buluoray requested review from a team and KiroCrewCaptain as code owners November 14, 2026 09:30
@buluoray
buluoray enabled auto-merge (squash) November 14, 2026 09:30
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 14, 2026
@github-actions

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ⏭️ skipped

Revision b9c137794f7c23fb3dc11d6c43ef34a147ca37a1 ships no reviewable capability (docs, tests or generated files only), so there is nothing to inventory. Does not block merge.

@github-actions

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed b9c137794f7c23fb3dc11d6c43ef34a147ca37a1 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] b9c1377

Verdict parsed from the review's SHA-scoped output markers for commit b9c137794f7c23fb3dc11d6c43ef34a147ca37a1.

False positive or not applicable? A repository writer can comment:
/ai-review override fable b9c137794f7c23fb3dc11d6c43ef34a147ca37a1 false-positive: <reason, 20 to 500 characters>

@github-actions

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision b9c137794f7c23fb3dc11d6c43ef34a147ca37a1 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of b9c137794f7c23fb3dc11d6c43ef34a147ca37a1 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: the change only updates two test expectations to match the kirocrew.step span that #37005 already merged.

[DESIGN-REVIEWED] b9c1377

Description read

[DESCRIPTION-READ] 5a91036f37fd4dd1a3544703915d716c66ce4474384156fdf3d38056a8342851

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of b9c137794f7c23fb3dc11d6c43ef34a147ca37a1 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] b9c1377

False positive or not applicable? A repository writer can comment:
/ai-review override gpt b9c137794f7c23fb3dc11d6c43ef34a147ca37a1 false-positive: <reason, 20 to 500 characters>

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified on head b9c1377. This changes test expectations only, in one file, test/tracing/test_tracing_otlp.py. The traced-turn test now expects kirocrew.step in its span set. The exit-flush test now expects 5 spans instead of 4. Both values match the step span that #37005 added in tracing/turns.py. They also match the five-span turn already pinned in test_tracing_turns.py and test_tracing_runs.py. No other assertion in the file still expects the old four-span set. The vault-header, redaction and loopback assertions are unchanged. The change adds no sleep, clock, port or ordering dependency.

Done well: the test still asserts the exact set of span names instead of a subset. So any span added later without a test update will still fail here.

@buluoray
buluoray merged commit 6de8528 into main Nov 14, 2026
98 checks passed
@buluoray
buluoray deleted the crew/slack/fix-otlp-step-span branch November 14, 2026 10:58
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Nov 14, 2026
chenmingwei23 pushed a commit that referenced this pull request Nov 15, 2026
#37005 adds one kirocrew.step span per derived model call to a traced
turn. The two wire-level OTLP tests still expected the four spans from
before, so they fail wherever the otlp extra is installed, which CI does
since #36721. Expect the step span and a count of five.

Closes #37169

(cherry picked from commit 6de8528)
chenmingwei23 pushed a commit that referenced this pull request Nov 15, 2026
…#37539

Two backend tests on release/0.13.0 were red for reasons main has
already fixed in test-only commits:

- test/tracing/test_tracing_otlp.py expected four spans, but #37005
  (already on the branch) adds kirocrew.step. Cherry-picked #37171
  unchanged.
- TestSelfModuleIndexIsLinear failed three runs in a row on the 0.5 s
  budget (0.522 s, 0.511 s, 0.509 s). Took main's end state of
  test/test_denied_commands_security_cost.py and
  test/denied_commands_security_helpers.py, which carries #37451 (min of
  five readings), #37530 (tracer paused) and #37539 (count work, not CPU
  time), plus the #37539 hunk of
  docs/system-specs/common/testing-conventions.md. The picks conflict
  one by one because the file also carries #28293, a test-only change.

No src/ changes.

(cherry picked from commit 6de8528)
buluoray added a commit that referenced this pull request Nov 16, 2026
…#37539

Two backend tests on release/0.13.0 were red for reasons main has
already fixed in test-only commits:

- test/tracing/test_tracing_otlp.py expected four spans, but #37005
  (already on the branch) adds kirocrew.step. Cherry-picked #37171
  unchanged.
- TestSelfModuleIndexIsLinear failed three runs in a row on the 0.5 s
  budget (0.522 s, 0.511 s, 0.509 s). Took main's end state of
  test/test_denied_commands_security_cost.py and
  test/denied_commands_security_helpers.py, which carries #37451 (min of
  five readings), #37530 (tracer paused) and #37539 (count work, not CPU
  time), plus the #37539 hunk of
  docs/system-specs/common/testing-conventions.md. The picks conflict
  one by one because the file also carries #28293, a test-only change.

No src/ changes.

(cherry picked from commit 6de8528)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

test: test_tracing_otlp expects four spans, but #37005 adds kirocrew.step (main red)

2 participants