Skip to content

feat(tracing): steps inside a turn: derived model calls, compaction, approvals, steers (#1643) - #37005

Merged
bolichen97 merged 1 commit into
mainfrom
crew/devmate/otel-pr4a
Nov 14, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
crew/devmate/otel-pr4a

Conversation

@kyleseaman

@kyleseaman kyleseaman commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: With tracing on, a turn's trace shows the steps inside it: each model call, any compaction the backend ran, each approval with how long it waited and who decided, and each steer that went in.

Today a traced turn is one invoke_agent span with a single aggregate chat span and its tool calls. A turn that waited twenty minutes on a person's approval looks the same as one whose model was slow. A turn that made five model calls shows none of the boundaries between them, and a steer or an in-turn compaction leaves no mark at all.

Why it matters

The tracing RFC's §5.3 lists approvals, model-call steps, steers and compaction as steps every turn's trace carries. Without them an operator reading a slow trace cannot tell whether the time went to a model, a tool, or a person who had not answered yet, or why a turn's context suddenly shrank.

Not a goal

  • Queue wait and the links a session_send, broadcast or workflow launch leave (the next PR, 4b).
  • Per-call usage: the harnesses report spend for the whole turn only, so no step carries any.
  • Approvals the harness decides itself: kiro-cli's auto-approvals never reach Crew.
  • Standalone compaction outside a turn (a later lifecycle phase), and new config keys or settings UI.

What changed (motivation → approach → change)

With tracing on, every traced turn gains child spans for the steps inside it.

Design record: the native tracing RFC §5.3 (steps), §5.8 (ids), §10 row 4. Earlier phases built the turn tree (tracing/turns.py) and the run roots; this is 4a.

Approach: what the event stream already shows is read where the turn's spans already watch the stream; the rest is reported by the Crew code that already handles it, through one new module, tracing/steps.py. Every step is a child of the turn span with ids H(kind, turn_id, attempt, n), and none carries spend.

  • Model-call steps. TurnTrace.observe opens a kirocrew.step at the turn's start and the next one at text after a tool group: the same boundary the dashboard's crew log derives, and the only per-call boundary the stream exposes. Each carries kirocrew.step.index and kirocrew.step.derived=true. Reading it from the stream gives every surface steps, not only the dashboard.
  • Compaction. The session's own compaction_status events give a kirocrew.compaction span from started to its result, with kirocrew.compaction.outcome; a fanned-out frame from another session is ignored, and a result with no start is a zero-length span.
  • Approvals. approval_opened / approval_closed wrap each prompt a person can answer: the dashboard's dashboard/chat_turn/permission_prompt.py::_on_permission_prompt, a channel's decider wait in messaging/driver.py::TurnDriver._approve (which every channel decider passes through), a sub-agent's tool prompt (_ApprovalPrompt, under the child's own turn) and its spawn approval (admission/pump.py, under the parent's spawn tool span). Each records kirocrew.approval.wait_ms, counted from when the prompt could first be answered (before a Slack or channel post that can be cancelled), decided_by (person; policy when the host decided with no answer, a cancelled wait included; timeout; unset when the sub-agent approver does not say who answered) and outcome. The close sits beside the existing crew-log decision, so every exit, cancel included, ends the span.
  • Steers. A zero-length kirocrew.steer with kirocrew.steer.mode (interrupt, follow_up), never the text: at the point the steer is written (acp/session_handle_steering.py::steer for kiro and Kiro Agent, acp/client_turn_control.py::steer for the direct clients), and for a sub-agent follow-up in follow_up_run_impl.
  • Allowlist: seven keys (kirocrew.step.index, .step.derived, .compaction.outcome, .steer.mode, .approval.wait_ms, .approval.decided_by, .approval.outcome).
  • Spec: metrics.md Traces, in this commit.

Rejected: emitting steps from chat_turn/stream_chunks.py, which only the dashboard runs; and tracing the channel approval at open_approval, which only the typed-reply channels call.

Backwards compatibility

Compatible: tracing is off by default, and off each new call returns at the switch read. Turned on, every traced turn has one more child span (its first kirocrew.step); a backend's span count per trace grows by that much.

Removes nothing: the diff adds spans and allowlisted attributes, and client_turn_control.steer sends each steer exactly as before.

Tests

test/tracing/test_tracing_steps.py (new):

  • Text after a tool group starts the next step; the steps hang under the turn, are marked derived, and carry no usage or Langfuse key while the turn keeps its credits.
  • An in-turn compaction is one span per result, failed is an error, another session's fanned-out frame is ignored, and no summary text rides a span.
  • An approval's wait equals the person's delay: the real channel _approve inside a traced channel turn, with a ManualClock, gives wait_ms == 7250, decided_by=person; an expired prompt gives timeout and rejected; an answer that landed while the prompt was still being posted counts from the post (wait_ms == 5000); a cancelled channel wait is policy. A numeric request id of 0 still gets its span.
  • A spawn approval sits under the parent's spawn tool span; a child's tool approval, driven through the real _ApprovalPrompt, under the child's own turn, with policy for a host answer and no decided_by for an unattributed one.
  • A steer is a zero-length span with its mode only; an unknown mode or a session with no open turn emits nothing.
  • On Claude, a steer held behind an open tool call is traced only when it is injected: one dropped at the turn's end, or still waiting, emits no span; an injected one emits exactly one.
  • The event kinds and the timeout cause spelled out in tracing/ match acp.types and constants.

test_off_imports_no_sdk_and_starts_no_thread now also calls the new approval and steer functions with tracing off. The pinned allowlist gains the seven keys, and four span-set pins gain the turn's kirocrew.step span.

Manual verification

N/A for manual steps: the tests drive the real channel approval ladder and the real turn trace with an in-memory exporter; no UI changes.

Local pre-push evidence, on main a67cbbb4d3: test/tracing plus the approval, steer, sub-agent approval and channel driver test files (2,588 tests) pass apart from test_client_bindings::test_every_class_resident_member_resolves_its_type_hints, which fails the same way on base under this worktree's Python 3.13. mypy on the whole package, flake8, isort, black, semgrep, the flake ratchet and the line cap pass.

Related Issues

Part of #1643 (phase 4a; follows #36717 and #36833).

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@kyleseaman
kyleseaman requested review from a team and KiroCrewCaptain as code owners November 14, 2026 01:43
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 14, 2026
@github-actions

github-actions Bot commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision d37d97bd57c73bdfe1ecef9eb4e16b165a20e8f3 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of d37d97bd57c73bdfe1ecef9eb4e16b165a20e8f3 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: tracing is off by default, every new hook is best-effort and catches its own errors, and the patch removes nothing.

Notes

  • Approval span ids are keyed by tool_call_id or approval_id rather than by request id. If one tool call gets asked twice (for example a retry after a timeout), both spans get the same span id.

[DESIGN-REVIEWED] d37d97b

Description read

[DESCRIPTION-READ] e184cdbe86cacd0aa1645d06dcc3ca847931e9aa1b6546ae9f50651881c804bc

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of d37d97bd57c73bdfe1ecef9eb4e16b165a20e8f3 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Nothing blocks. Every step is listed in RFC §5.3 (base status partial), tracing is off by default, and none of the items duplicates an existing span or attribute.

Notes

  • Item 4 — inherited: RFC §5.3 lists steer under "Links (not parents) for asynchronous causes: session_send and broadcast, steer, …", but the diff adds a child span instead: turn.numbered(SPAN_STEER, …) and then span.end(end_time=span.start_time). The description gives no reason for leaving the RFC's design. Subtraction: defer steered() and its four call sites to 4b, where the RFC's link work is planned, rather than shipping a second form of steer now.

What this change ships

Inventory (7 items) — 6 justified

Intent: an operator reading a slow trace can tell whether the time went to the model, a tool, a person's approval, a compaction or a steer. ADDITION.

  1. Each traced turn shows one kirocrew.step span per derived model call — justified
  2. A compaction the backend runs during a turn appears as a kirocrew.compaction span with its outcome — justified
  3. Each approval a person can answer (dashboard, channel, sub-agent tool, spawn) appears as a span with its wait time, who decided, and the outcome — justified
  4. Each steer appears as a zero-length kirocrew.steer span carrying its mode — inherited: the RFC says steer is a link
  5. Seven new allowlisted span attributes — justified
  6. provider.start_span takes a backdated start_time — justified
  7. client_turn_control.steer now has one exit, so the span is emitted after a send — justified

[FIRST-PRINCIPLES-REVIEWED] d37d97b

Description read

[DESCRIPTION-READ] 2e1826a07e0c884b8e7d5ecfc01c88a2cf438cc787bef882e494508f2b7df401

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed d37d97bd57c73bdfe1ecef9eb4e16b165a20e8f3 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] d37d97b

Verdict parsed from the review's SHA-scoped output markers for commit d37d97bd57c73bdfe1ecef9eb4e16b165a20e8f3.

False positive or not applicable? A repository writer can comment:
/ai-review override fable d37d97bd57c73bdfe1ecef9eb4e16b165a20e8f3 false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of d37d97bd57c73bdfe1ecef9eb4e16b165a20e8f3 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] d37d97b

False positive or not applicable? A repository writer can comment:
/ai-review override gpt d37d97bd57c73bdfe1ecef9eb4e16b165a20e8f3 false-positive: <reason, 20 to 500 characters>

@KiroCrewCaptain

KiroCrewCaptain commented Nov 14, 2026 •

Copy link
Copy Markdown
Collaborator

Code change approver recommendation: A person should decide

Why: the decider thinks the change is probably fine but estimates a 14% chance that this change introduces a defect, above the 12% limit for approving on its own.

Tracing-only addition, off by default: step, compaction, approval and steer spans under each traced turn. Verified: (1) client_turn_control.steer's single-exit rewrite keeps every branch's behaviour; the Claude held-steer path emits its span only on injection in _send_steering_request, and the client and session-handle steers are separate paths, so one steer never gets two spans. (2) Every approval span (dashboard, channel _approve, spawn pump, sub-agent _ApprovalPrompt via CallbackResponder) closes in a finally beside the existing crew-log decision, cancels included, and decided_by follows each site's attribution. (3) TurnTrace ends the open step and compaction at finish; a steer after the turn ends finds no open turn; new spans count toward the span budget and go through the attribute allowlist. (4) Tests drive the real channel ladder, the sub-agent prompt watch and the Claude held-steer paths. No regression, data-loss or trust-boundary issue found.

An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit d37d97bd57 · 881 changed lines. If this recommendation is wrong, reply here: replies are how the approver is evaluated.

@kyleseaman
kyleseaman force-pushed the crew/devmate/otel-pr4a branch from c15c450 to 8044d0d Compare November 14, 2026 02:51
@kyleseaman

kyleseaman commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor Author

self-added: yes
mechanism: approval span id read from a str|int request id

  • span=1ead1f49a74d Numeric request ID zero suppresses channel approval tracing

fixed in 8044d0d: TurnDriver._approve now passes str(event.request_id) whenever the id is not None, so a JSON-RPC id of 0 opens its span as "0"; a missing or None id still gives "".
Regression: test/tracing/test_tracing_steps.py::test_a_channel_approval_with_request_id_zero_exports_one_span drives the real _approve with request_id=0 and fails without the fix.
Swept the other approval call sites in this diff: permission_prompt.py passes str(event.request_id); pump.py and run.py pass string ids, so none share the falsy-id read.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 14, 2026
@kyleseaman
kyleseaman force-pushed the crew/devmate/otel-pr4a branch from 8044d0d to c2f3ffd Compare November 14, 2026 03:58
@kyleseaman

Copy link
Copy Markdown
Contributor Author

self-added: yes
mechanism: steer span emitted where a steer is accepted locally

  • span=9ade13dd3820 Held Claude steers are traced before delivery

fixed in c2f3ffd: steer() no longer emits the span on the ACP_BACKENDS_STEERING_REQUEST branch; _send_steering_request emits it where the adapter answers injected for the still-running turn, covering both the immediate and the held send.
Regression: test/tracing/test_tracing_steps.py drives a real Claude AcpClient: a held steer dropped at the turn's end, or still waiting, emits no span; an injected one emits exactly one. Two of them fail without the fix.
Swept the other trace_steps.steered( sites: the session handle's codex and kiro paths emit only after the answer or write, and a sub-agent follow-up records its queueing by design.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Nov 14, 2026
@kyleseaman
kyleseaman force-pushed the crew/devmate/otel-pr4a branch from c2f3ffd to d37d97b Compare November 14, 2026 05:58
@kyleseaman

Copy link
Copy Markdown
Contributor Author

Rebased: main moved and fixed a red this head carried (c2f3ffd -> d37d97b, base f9ebf3a -> a67cbbb); CI re-running. test/test_search_docs.py::test_search_docs_ranks_the_page_the_query_is_about_first failed on base f9ebf3a and passes on main, fixed by #37063; a rerun reuses the old merge commit, so only a new head clears it. No change to this PR's own diff.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Nov 14, 2026

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I read the whole diff on this head, and all checks are green.

  • This adds trace spans inside a turn, per the native tracing RFC §5.3: approval waits at the four places a person can answer (dashboard prompt, channel decider, sub-agent tool prompt, spawn approval), and steers as point spans. A steer span carries only its mode, never the text, and approval spans carry ids, wait time, who decided and the outcome, not the tool arguments.
  • Every entry point in tracing/steps.py (approval_opened, approval_closed, steered) wraps its body in try/except and never awaits, so tracing cannot change or fail a turn. The call sites keep main's outcomes:
    • TurnDriver._approve returns the decider's answer as before. The new try/finally only closes the span, and a raise or cancel still propagates.
    • AcpClient.steer still returns the same sent on every backend branch. The held-behind-a-tool-call branch keeps its early return, and its span is written where the steer is injected.
    • The dashboard, sub-agent and spawn sites close the span beside the existing crew-log decision, so every exit, cancel included, ends it.
  • No request is refused, no shared read API changes shape, and no regex is added.

Approving.

@bolichen97
bolichen97 merged commit 12a03ac into main Nov 14, 2026
103 of 105 checks passed
@bolichen97
bolichen97 deleted the crew/devmate/otel-pr4a branch November 14, 2026 08:30
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Nov 14, 2026
buluoray added a commit that referenced this pull request Nov 14, 2026
#37005 adds one kirocrew.step span per derived model call to a traced
turn. The two wire-level OTLP tests still expected the four spans from
before, so they fail wherever the otlp extra is installed, which CI does
since #36721. Expect the step span and a count of five.

Closes #37169
buluoray pushed a commit that referenced this pull request Nov 15, 2026
…approvals, steers (#1643) (#37005)

Co-authored-by: Kyle Seaman <kseam@dev-dsk-kseam-1b-55230d27.us-east-1.amazon.com>
(cherry picked from commit 12a03ac)
chenmingwei23 pushed a commit that referenced this pull request Nov 15, 2026
#37005 adds one kirocrew.step span per derived model call to a traced
turn. The two wire-level OTLP tests still expected the four spans from
before, so they fail wherever the otlp extra is installed, which CI does
since #36721. Expect the step span and a count of five.

Closes #37169

(cherry picked from commit 6de8528)
chenmingwei23 pushed a commit that referenced this pull request Nov 15, 2026
…#37539

Two backend tests on release/0.13.0 were red for reasons main has
already fixed in test-only commits:

- test/tracing/test_tracing_otlp.py expected four spans, but #37005
  (already on the branch) adds kirocrew.step. Cherry-picked #37171
  unchanged.
- TestSelfModuleIndexIsLinear failed three runs in a row on the 0.5 s
  budget (0.522 s, 0.511 s, 0.509 s). Took main's end state of
  test/test_denied_commands_security_cost.py and
  test/denied_commands_security_helpers.py, which carries #37451 (min of
  five readings), #37530 (tracer paused) and #37539 (count work, not CPU
  time), plus the #37539 hunk of
  docs/system-specs/common/testing-conventions.md. The picks conflict
  one by one because the file also carries #28293, a test-only change.

No src/ changes.

(cherry picked from commit 6de8528)
buluoray added a commit that referenced this pull request Nov 16, 2026
…#37539

Two backend tests on release/0.13.0 were red for reasons main has
already fixed in test-only commits:

- test/tracing/test_tracing_otlp.py expected four spans, but #37005
  (already on the branch) adds kirocrew.step. Cherry-picked #37171
  unchanged.
- TestSelfModuleIndexIsLinear failed three runs in a row on the 0.5 s
  budget (0.522 s, 0.511 s, 0.509 s). Took main's end state of
  test/test_denied_commands_security_cost.py and
  test/denied_commands_security_helpers.py, which carries #37451 (min of
  five readings), #37530 (tracer paused) and #37539 (count work, not CPU
  time), plus the #37539 hunk of
  docs/system-specs/common/testing-conventions.md. The picks conflict
  one by one because the file also carries #28293, a test-only change.

No src/ changes.

(cherry picked from commit 6de8528)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants