Repository navigation
feat(tracing): steps inside a turn: derived model calls, compaction, approvals, steers (#1643) - #37005
Conversation
UX Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing blocks: tracing is off by default, every new hook is best-effort and catches its own errors, and the patch removes nothing. Notes
[DESIGN-REVIEWED] d37d97b Description read[DESCRIPTION-READ] e184cdbe86cacd0aa1645d06dcc3ca847931e9aa1b6546ae9f50651881c804bc That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Nothing blocks. Every step is listed in RFC §5.3 (base status Notes
What this change shipsInventory (7 items) — 6 justifiedIntent: an operator reading a slow trace can tell whether the time went to the model, a tool, a person's approval, a compaction or a steer. ADDITION.
[FIRST-PRINCIPLES-REVIEWED] d37d97b Description read[DESCRIPTION-READ] 2e1826a07e0c884b8e7d5ecfc01c88a2cf438cc787bef882e494508f2b7df401 That sha256 names the PR description this verdict read. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
Code change approver recommendation: A person should decideWhy: the decider thinks the change is probably fine but estimates a 14% chance that this change introduces a defect, above the 12% limit for approving on its own. Tracing-only addition, off by default: step, compaction, approval and steer spans under each traced turn. Verified: (1) client_turn_control.steer's single-exit rewrite keeps every branch's behaviour; the Claude held-steer path emits its span only on injection in _send_steering_request, and the client and session-handle steers are separate paths, so one steer never gets two spans. (2) Every approval span (dashboard, channel _approve, spawn pump, sub-agent _ApprovalPrompt via CallbackResponder) closes in a finally beside the existing crew-log decision, cancels included, and decided_by follows each site's attribution. (3) TurnTrace ends the open step and compaction at finish; a steer after the turn ends finds no open turn; new spans count toward the span budget and go through the attribute allowlist. (4) Tests drive the real channel ladder, the sub-agent prompt watch and the Claude held-steer paths. No regression, data-loss or trust-boundary issue found. An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit |
c15c450 to
8044d0d
Compare
|
self-added: yes
|
8044d0d to
c2f3ffd
Compare
|
self-added: yes
|
…approvals, steers (#1643)
c2f3ffd to
d37d97b
Compare
|
Rebased: main moved and fixed a red this head carried (c2f3ffd -> d37d97b, base f9ebf3a -> a67cbbb); CI re-running. |
bolichen97
left a comment
There was a problem hiding this comment.
I read the whole diff on this head, and all checks are green.
- This adds trace spans inside a turn, per the native tracing RFC §5.3: approval waits at the four places a person can answer (dashboard prompt, channel decider, sub-agent tool prompt, spawn approval), and steers as point spans. A steer span carries only its mode, never the text, and approval spans carry ids, wait time, who decided and the outcome, not the tool arguments.
- Every entry point in
tracing/steps.py(approval_opened,approval_closed,steered) wraps its body intry/exceptand never awaits, so tracing cannot change or fail a turn. The call sites keep main's outcomes:TurnDriver._approvereturns the decider's answer as before. The newtry/finallyonly closes the span, and a raise or cancel still propagates.AcpClient.steerstill returns the samesenton every backend branch. The held-behind-a-tool-call branch keeps its early return, and its span is written where the steer is injected.- The dashboard, sub-agent and spawn sites close the span beside the existing crew-log decision, so every exit, cancel included, ends it.
- No request is refused, no shared read API changes shape, and no regex is added.
Approving.
#37005 adds one kirocrew.step span per derived model call to a traced turn. The two wire-level OTLP tests still expected the four spans from before, so they fail wherever the otlp extra is installed, which CI does since #36721. Expect the step span and a count of five. Closes #37169 (cherry picked from commit 6de8528)
…#37539 Two backend tests on release/0.13.0 were red for reasons main has already fixed in test-only commits: - test/tracing/test_tracing_otlp.py expected four spans, but #37005 (already on the branch) adds kirocrew.step. Cherry-picked #37171 unchanged. - TestSelfModuleIndexIsLinear failed three runs in a row on the 0.5 s budget (0.522 s, 0.511 s, 0.509 s). Took main's end state of test/test_denied_commands_security_cost.py and test/denied_commands_security_helpers.py, which carries #37451 (min of five readings), #37530 (tracer paused) and #37539 (count work, not CPU time), plus the #37539 hunk of docs/system-specs/common/testing-conventions.md. The picks conflict one by one because the file also carries #28293, a test-only change. No src/ changes. (cherry picked from commit 6de8528)
…#37539 Two backend tests on release/0.13.0 were red for reasons main has already fixed in test-only commits: - test/tracing/test_tracing_otlp.py expected four spans, but #37005 (already on the branch) adds kirocrew.step. Cherry-picked #37171 unchanged. - TestSelfModuleIndexIsLinear failed three runs in a row on the 0.5 s budget (0.522 s, 0.511 s, 0.509 s). Took main's end state of test/test_denied_commands_security_cost.py and test/denied_commands_security_helpers.py, which carries #37451 (min of five readings), #37530 (tracer paused) and #37539 (count work, not CPU time), plus the #37539 hunk of docs/system-specs/common/testing-conventions.md. The picks conflict one by one because the file also carries #28293, a test-only change. No src/ changes. (cherry picked from commit 6de8528)
Problem / Motivation
Goal: With tracing on, a turn's trace shows the steps inside it: each model call, any compaction the backend ran, each approval with how long it waited and who decided, and each steer that went in.
Today a traced turn is one
invoke_agentspan with a single aggregatechatspan and its tool calls. A turn that waited twenty minutes on a person's approval looks the same as one whose model was slow. A turn that made five model calls shows none of the boundaries between them, and a steer or an in-turn compaction leaves no mark at all.Why it matters
The tracing RFC's §5.3 lists approvals, model-call steps, steers and compaction as steps every turn's trace carries. Without them an operator reading a slow trace cannot tell whether the time went to a model, a tool, or a person who had not answered yet, or why a turn's context suddenly shrank.
Not a goal
session_send, broadcast or workflow launch leave (the next PR, 4b).What changed (motivation → approach → change)
With tracing on, every traced turn gains child spans for the steps inside it.
Design record: the native tracing RFC §5.3 (steps), §5.8 (ids), §10 row 4. Earlier phases built the turn tree (
tracing/turns.py) and the run roots; this is 4a.Approach: what the event stream already shows is read where the turn's spans already watch the stream; the rest is reported by the Crew code that already handles it, through one new module,
tracing/steps.py. Every step is a child of the turn span with idsH(kind, turn_id, attempt, n), and none carries spend.TurnTrace.observeopens akirocrew.stepat the turn's start and the next one at text after a tool group: the same boundary the dashboard's crew log derives, and the only per-call boundary the stream exposes. Each carrieskirocrew.step.indexandkirocrew.step.derived=true. Reading it from the stream gives every surface steps, not only the dashboard.compaction_statusevents give akirocrew.compactionspan fromstartedto its result, withkirocrew.compaction.outcome; a fanned-out frame from another session is ignored, and a result with no start is a zero-length span.approval_opened/approval_closedwrap each prompt a person can answer: the dashboard'sdashboard/chat_turn/permission_prompt.py::_on_permission_prompt, a channel's decider wait inmessaging/driver.py::TurnDriver._approve(which every channel decider passes through), a sub-agent's tool prompt (_ApprovalPrompt, under the child's own turn) and its spawn approval (admission/pump.py, under the parent's spawn tool span). Each recordskirocrew.approval.wait_ms, counted from when the prompt could first be answered (before a Slack or channel post that can be cancelled),decided_by(person;policywhen the host decided with no answer, a cancelled wait included;timeout; unset when the sub-agent approver does not say who answered) andoutcome. The close sits beside the existing crew-log decision, so every exit, cancel included, ends the span.kirocrew.steerwithkirocrew.steer.mode(interrupt,follow_up), never the text: at the point the steer is written (acp/session_handle_steering.py::steerfor kiro and Kiro Agent,acp/client_turn_control.py::steerfor the direct clients), and for a sub-agent follow-up infollow_up_run_impl.kirocrew.step.index,.step.derived,.compaction.outcome,.steer.mode,.approval.wait_ms,.approval.decided_by,.approval.outcome).Rejected: emitting steps from
chat_turn/stream_chunks.py, which only the dashboard runs; and tracing the channel approval atopen_approval, which only the typed-reply channels call.Backwards compatibility
Compatible: tracing is off by default, and off each new call returns at the switch read. Turned on, every traced turn has one more child span (its first
kirocrew.step); a backend's span count per trace grows by that much.Removes nothing: the diff adds spans and allowlisted attributes, and
client_turn_control.steersends each steer exactly as before.Tests
test/tracing/test_tracing_steps.py(new):failedis an error, another session's fanned-out frame is ignored, and no summary text rides a span._approveinside a traced channel turn, with aManualClock, giveswait_ms == 7250,decided_by=person; an expired prompt givestimeoutandrejected; an answer that landed while the prompt was still being posted counts from the post (wait_ms == 5000); a cancelled channel wait ispolicy. A numeric request id of 0 still gets its span._ApprovalPrompt, under the child's own turn, withpolicyfor a host answer and nodecided_byfor an unattributed one.tracing/matchacp.typesandconstants.test_off_imports_no_sdk_and_starts_no_threadnow also calls the new approval and steer functions with tracing off. The pinned allowlist gains the seven keys, and four span-set pins gain the turn'skirocrew.stepspan.Manual verification
N/A for manual steps: the tests drive the real channel approval ladder and the real turn trace with an in-memory exporter; no UI changes.
Local pre-push evidence, on main
a67cbbb4d3:test/tracingplus the approval, steer, sub-agent approval and channel driver test files (2,588 tests) pass apart fromtest_client_bindings::test_every_class_resident_member_resolves_its_type_hints, which fails the same way on base under this worktree's Python 3.13. mypy on the whole package, flake8, isort, black, semgrep, the flake ratchet and the line cap pass.Related Issues
Part of #1643 (phase 4a; follows #36717 and #36833).
Checklist