Repository navigation
feat(tracing): an Ultra Mode round as one trace, with a span per ledger write (5a, #1643) - #36833
Conversation
UX Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing blocks: this is tracing that is off by default, the ledger keys it adds only feed traces, and a round's trace ids are derived from the ledger itself, so nothing is held open across a restart. Notes
[DESIGN-REVIEWED] 48c3ad7 Description read[DESCRIPTION-READ] 2ed0b5953528ddb633c506e8b2747bee4828d8863bdc05c4f837d06007df5c9b That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Every item maps to accepted RFC rfc-otel-tracing §5.5 and §10 row 5. That covers the new What this change shipsInventory (9 items) — 9 justifiedIntent: with tracing on, an operator sees one Ultra Mode or Bot round as one waterfall trace (its turns, items, ledger writes and outcomes). This is an ADDITION.
[FIRST-PRINCIPLES-REVIEWED] 48c3ad7 Description read[DESCRIPTION-READ] 30628b5f7be5020fd9fb189e52f050d78f74433a9375670d2a6d6c0b7dd2f84f That sha256 names the PR description this verdict read. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
Code change approver recommendation: A person should decideWhy: the decider thinks the change is probably fine but estimates a 20% chance that this change introduces a defect, above the 12% limit for approving on its own. This PR adds tracing of Ultra Mode and Bot rounds; tracing is off by default. Every new hook returns once provider.peek() reports tracing off. The ledger change is two header keys (goal_id, round_started_at), written under the existing lock and snapshot, so an undo restores them; a rebuild keeps them, and the 1 MB production record cap is unaffected. Round and item ids, and the roll-up totals, come out the same on the write routes and in turn binding, and the follow-up step after a turn is added under a lock, so it is neither lost nor run twice. Every route that writes to the ledger calls recorded only after its crew-log entry landed. Span attributes are allowlisted ids, fixed-vocabulary values and counts, run through the existing cleaning and redaction. The new tests drive the real routes and turn binding across round changes, rollover, restart, rebuild, turn mode, tracing off and a no-free-text check. The remaining gaps (anchors missing until the next goal write on old boards, thin tests for two routes) are known and non-blocking. An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit |
6430b78 to
48c3ad7
Compare
|
self-added: yes
|
|
self-added: yes
|
bolichen97
left a comment
There was a problem hiding this comment.
Read the whole diff (24 files, 1878 lines). This is RFC section 10 row 5 (phase 5a) of the tracing RFC that its owner accepted in full, the same RFC as phase 3a. The RFC's own text names every piece this PR adds: the round and item anchors and outcomes, goal_id on the ledger header, the ledger-write spans, and the traces_ultra_mode setting (Q9, default one trace per round, turn keeps per-turn traces).
Checked:
- Off path. With
telemetry.traces_enabledoff (the default),header_beforereads nothing,recordedreturns onprovider.peek() is None, andbindingreturns(None, ""). A test pins that nothing is read or emitted. - Data leaving the host. The new span attributes are ids and closed vocabularies (action, actor, state, verdict, status, reason, counts, a PR number).
_write_attrsruns_enumover every value againstwork_vocab, so free text cannot pass, and a test feeds a sentinel to every free-text field and checks that no span carries it. No new export or destination. - The one change that runs with tracing off is the ledger header:
goal_idandround_started_atare minted by the ledger.from_dictreads an older header without them, a rebuild keeps the cache's values, and the record cap test moves from 360 to 440 bytes only because the header grew. No crew-log entry changes. - No new refusal, and no regex in product code.
- The RFC edits (outcome named
ultra outcome <closed|advanced>, item anchor keyed by goal and round) record what the code does and stay inside the accepted design.
The shortcut comments (no anchor for a round that began while tracing was off, a goal's last round has no outcome) are stated in the code and the metrics doc. 97 checks pass, 0 red, 0 pending.
Problem / Motivation
Goal: With tracing on, one round of an Ultra Mode or Bot conductor's work shows up as one trace: the conductor's turns, one span per work item holding its crew member's turns, a span for every ledger write, and how each item and the round ended.
Today every conductor and worker turn is its own trace. A round with three items and a dozen turns is a dozen unrelated traces, held together only by the
kirocrew.crew.rootandkirocrew.work.item_idattributes. Ledger writes (create, bind, verdict, close, reports) leave no span at all, so a trace cannot show when an item was dispatched, reported on or accepted.Why it matters
Ultra Mode is where one request fans out into the most sessions and the most spend. The tracing RFC (§5.5) chose one trace per round so a round reads as one waterfall in Langfuse or X-Ray, the way a workflow run does. Without it an operator cannot see which item a slow or expensive turn served, what an item cost, or how long it took from dispatch to acceptance.
Not a goal
What changed (motivation → approach → change)
Design record: the native tracing RFC §5.5 and §10 row 5. This is 5a; the RFC's ~950-line phase is split in two so each half stays reviewable.
Approach: a round is one more run kind on 1b's run-root helpers (
tracing/runs.py): zero-length anchors that children find by deterministic id, so nothing is held open and a restart re-derives every id.ultra round N(conductor, goal_id, round); links to the previous round. The conductor's turns sit under it.work_item <item id>kirocrew.work.<action>work_item outcome <state>close, with verdict, fails and the item's summed spend.ultra outcome closed|advancedclosedwith no item open,advancedwith items carried over. Closing the last item does not end it, since a new item can follow in the same round.goal_idwork_ledger_records.py), minted with the board and by eachnew_goal(work_ledger_writes.py), so rounds of one goal sharekirocrew.work.goal_id. A rebuild keeps it (src/kiro_crew/work_ledger.py). A second new key,round_started_at, stamps when the round began, so the age rollover holds across a restart.telemetry.traces_ultra_moderun(default) orturn(config/sections_telemetry.py):turnkeeps one trace per turn, with the write spans inside each turn.trace_binding.bind_turn, off the loop. A turn a finished sub-agent run caused keeps that join.closeor round change usually happens inside a conductor turn that has not added its spend yet, so the outcome marker is written when that turn ends (turns.after_turn, one new function and one call inTurnTrace.finish).round_started_at, moves to oneultra round N (cont 2)trace. Each item's anchor is emitted again under it on first use, and outcomes and turnless writes follow it. A round's roll-up is keyed by its first anchor and an item's by the item itself, so an item's outcome sums every turn it took, across round changes and rollovers.on_work_recordedcall:work_ledger_conductor.py(every conductor action, remote and hand-back ones included),work_ledger_worker.py(report, person steer) andremote_ingest.py(a peer's report). Off, each costs onepeek().config-baseline.jsonregenerated for the new key.Rejected: holding a round span open in memory (lost on restart, and a round can run for days), and one trace per goal (a goal can outlive X-Ray's 7-day window).
Backwards compatibility
Compatible: tracing is off by default, and off, the routes add one
peek()each and read nothing.goal_idis a new header key; a header from before it reads empty and stands in with<generation>.<goal_items_base>until its next goal write mints one.round_started_atis new too; a header without it reads 0, so only the span budget rolls such a round over until its next round change.runs.outcometakes an optionalattrs=, the same signature #36717 adds.Changed for tracing users: a turn on an Ultra Mode or Bot board now sits in its round's trace instead of rooting its own, unless
traces_ultra_modeisturn.Removes nothing: the diff adds spans, one header key, one config key and allowlisted attributes.
Tests
test/tracing/test_tracing_work.py(new, 18 tests), driving the realapi_work_ledger_record/api_work_reportroutes on an isolated data home and binding every turn through the realtrace_binding.bind_turn:goal_id. In each: the conductor's turns under the round,createandbindspans in the conductor turn that made them, the worker turn under its item, itsreportspan inside that turn,work_item outcome acceptedwith verdict and the worker's 2.5 credits,ultra outcome closed, and the conductor'ssession.idon every span.goalandcreatespans under the round they open, linked to the turn.(cont 2)round: the item's anchor, both worker turns and the item's outcome are there, and the outcome sums the spend from before and after the rollover.goal_id.(cont 2)trace.advanced.goal_id;new_goalmints a new one; a rebuild from the crew log keeps it.turnmode emits no anchors, roots every turn, and keeps the write span in its turn.root_kind=mate.(cont 2)trace.test_tracing_ids_attrs.py: the pinned allowlist gains the tenkirocrew.work.*keys.test_work_ledger_caps.py's oversized-acceptance test raises its small record cap to 440 bytes, above the header's new 394.test_work_ledger.pynamestracing/work.pyas a store reader in its seam allowlist, with the same argument astracing/crew.py.Determinism: 20 runs at
-n0and 10 at-n 4 --dist loadgroup --max-worker-restart=2besidetest_tracing_runs.pyandtest_tracing_surfaces.py, all green; a shuffled order (seed 214) green. No sleeps: every wait is an awaited route call.Local: related backend tests (11,530 passed; the 47 red ones, in
file_explorer,aws_controlpackaging and two others, fail the same on a cleanorigin/mainworktree on this host), semgrep rules on the diff (0 findings), the flake-pattern ratchet and the pre-push gates pass.Manual verification
N/A — unit coverage drives the real routes and turn binding; no UI or live collector is involved.
Follow-ups
doneturn has ended. Fixing it means also waiting on the worker's open turn.ultra outcomemarker, and keeps its roll-up entry, until a later goal write moves the board on: the ledger has no "goal finished" action to end it on. The item outcomes and every turn are still in the trace. Ending it needs a ledger signal for a finished goal, or ending the round when its last item closes in a way a later create can reopen.(cont 2)anchor again, with the same ids. Storing the live continuation on the header would stop that.Related Issues
Part of #1643
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)