Skip to content

feat(tracing): an Ultra Mode round as one trace, with a span per ledger write (5a, #1643) - #36833

Merged
bolichen97 merged 1 commit into
mainfrom
crew/devmate/otel-pr5
Nov 14, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
crew/devmate/otel-pr5

Conversation

@kyleseaman

@kyleseaman kyleseaman commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: With tracing on, one round of an Ultra Mode or Bot conductor's work shows up as one trace: the conductor's turns, one span per work item holding its crew member's turns, a span for every ledger write, and how each item and the round ended.

Today every conductor and worker turn is its own trace. A round with three items and a dozen turns is a dozen unrelated traces, held together only by the kirocrew.crew.root and kirocrew.work.item_id attributes. Ledger writes (create, bind, verdict, close, reports) leave no span at all, so a trace cannot show when an item was dispatched, reported on or accepted.

Why it matters

Ultra Mode is where one request fans out into the most sessions and the most spend. The tracing RFC (§5.5) chose one trace per round so a round reads as one waterfall in Langfuse or X-Ray, the way a workflow run does. Without it an operator cannot see which item a slow or expensive turn served, what an item cost, or how long it took from dispatch to acceptance.

Not a goal

  • Worker creation spans, report → wake links, undelivered-report events, manager-change spans (handoff, adopt, release, hand-back) and patrol health counts. Those are phase 5b, a follow-up PR on top of this one.
  • Any change to what the work ledger does, refuses or records in the crew log.
  • New export destinations or settings UI.

What changed (motivation → approach → change)

Design record: the native tracing RFC §5.5 and §10 row 5. This is 5a; the RFC's ~950-line phase is split in two so each half stays reviewable.

Approach: a round is one more run kind on 1b's run-root helpers (tracing/runs.py): zero-length anchors that children find by deterministic id, so nothing is held open and a restart re-derives every id.

Piece What it does
ultra round N anchor of one round, keyed (conductor, goal_id, round); links to the previous round. The conductor's turns sit under it.
work_item <item id> anchor per item in the round. The bound worker's turns sit under it. An item still open when the round moves on gets an anchor in the new round too.
kirocrew.work.<action> one span per ledger write whose crew-log entry landed, inside the acting turn when that turn is in the write's round. Otherwise under the item's or round's anchor, linked to the acting turn if there is one: the turn that opens a round was bound before it, and a person's steer or a remote crew's report has no local turn.
work_item outcome <state> written by close, with verdict, fails and the item's summed spend.
ultra outcome closed|advanced the round ends once, at the goal write that moves the board on: closed with no item open, advanced with items carried over. Closing the last item does not end it, since a new item can follow in the same round.
goal_id a new random id on the ledger header (work_ledger_records.py), minted with the board and by each new_goal (work_ledger_writes.py), so rounds of one goal share kirocrew.work.goal_id. A rebuild keeps it (src/kiro_crew/work_ledger.py). A second new key, round_started_at, stamps when the round began, so the age rollover holds across a restart.
telemetry.traces_ultra_mode run (default) or turn (config/sections_telemetry.py): turn keeps one trace per turn, with the write spans inside each turn.
  • Where anchors come from. Only the write that brings a key into being emits one: the board's first write, a goal write that changes the goal or round, and a create. So nothing is emitted twice, and a turn finds its anchor by reading the header (a worker through its binding) in trace_binding.bind_turn, off the loop. A turn a finished sub-agent run caused keeps that join.
  • Outcomes wait for the acting turn. A close or round change usually happens inside a conductor turn that has not added its spend yet, so the outcome marker is written when that turn ends (turns.after_turn, one new function and one call in TurnTrace.finish).
  • Rollover. A round past R5's span budget, or six days old by round_started_at, moves to one ultra round N (cont 2) trace. Each item's anchor is emitted again under it on first use, and outcomes and turnless writes follow it. A round's roll-up is keyed by its first anchor and an item's by the item itself, so an item's outcome sums every turn it took, across round changes and rollovers.
  • Where write spans come from. Next to each on_work_recorded call: work_ledger_conductor.py (every conductor action, remote and hand-back ones included), work_ledger_worker.py (report, person steer) and remote_ingest.py (a peer's report). Off, each costs one peek().
  • Content. Attributes are ids, closed-vocabulary values and counts: action, actor, item id, round, generation, goal id, state, verdict, fails, report status and reason, PR number. Never the goal, a title, acceptance, decision, summary or artifact text.
  • Specs: metrics.md Traces, the tracing guide, the RFC's id rows, and the work-ledger RFC's header table, in this commit. config-baseline.json regenerated for the new key.

Rejected: holding a round span open in memory (lost on restart, and a round can run for days), and one trace per goal (a goal can outlive X-Ray's 7-day window).

Backwards compatibility

Compatible: tracing is off by default, and off, the routes add one peek() each and read nothing. goal_id is a new header key; a header from before it reads empty and stands in with <generation>.<goal_items_base> until its next goal write mints one. round_started_at is new too; a header without it reads 0, so only the span budget rolls such a round over until its next round change. runs.outcome takes an optional attrs=, the same signature #36717 adds.

Changed for tracing users: a turn on an Ultra Mode or Bot board now sits in its round's trace instead of rooting its own, unless traces_ultra_mode is turn.

Removes nothing: the diff adds spans, one header key, one config key and allowlisted attributes.

Tests

test/tracing/test_tracing_work.py (new, 18 tests), driving the real api_work_ledger_record / api_work_report routes on an isolated data home and binding every turn through the real trace_binding.bind_turn:

  • A two-item, two-round replay (then a goal write that moves on) gives two round traces, linked, sharing one goal_id. In each: the conductor's turns under the round, create and bind spans in the conductor turn that made them, the worker turn under its item, its report span inside that turn, work_item outcome accepted with verdict and the worker's 2.5 credits, ultra outcome closed, and the conductor's session.id on every span.
  • The turn that bootstraps a board, and the turn that moves it to round 2, put their goal and create spans under the round they open, linked to the turn.
  • A round where the last item closes and a new item follows ends once, at the next goal write, and its roll-up has all three conductor turns' credits, the closing turn's included.
  • A round past its span budget keeps its item under the (cont 2) round: the item's anchor, both worker turns and the item's outcome are there, and the outcome sums the spend from before and after the rollover.
  • A rebuild of a dirty cache keeps goal_id.
  • An item open across a round change sums the spend from both rounds in its outcome.
  • After a restart, a round older than the six-day age budget moves to a (cont 2) trace.
  • A restart (in-memory registries cleared) re-derives the same anchor ids, and the anchor is emitted once.
  • An item open across a round change moves to the new round, and the old round ends advanced.
  • A reword keeps goal_id; new_goal mints a new one; a rebuild from the crew log keeps it.
  • turn mode emits no anchors, roots every turn, and keeps the write span in its turn.
  • A write no local turn made sits under its item, with the raw item id.
  • A Bot board traces as root_kind=mate.
  • A round past its span budget rolls over to a linked (cont 2) trace.
  • Off: nothing is read and nothing is emitted.
  • Free-text sentinels fed to the goal, title, decision, summary and artifacts appear in no span name or attribute.

test_tracing_ids_attrs.py: the pinned allowlist gains the ten kirocrew.work.* keys. test_work_ledger_caps.py's oversized-acceptance test raises its small record cap to 440 bytes, above the header's new 394. test_work_ledger.py names tracing/work.py as a store reader in its seam allowlist, with the same argument as tracing/crew.py.

Determinism: 20 runs at -n0 and 10 at -n 4 --dist loadgroup --max-worker-restart=2 beside test_tracing_runs.py and test_tracing_surfaces.py, all green; a shuffled order (seed 214) green. No sleeps: every wait is an awaited route call.

Local: related backend tests (11,530 passed; the 47 red ones, in file_explorer, aws_control packaging and two others, fail the same on a clean origin/main worktree on this host), semgrep rules on the diff (0 findings), the flake-pattern ratchet and the pre-push gates pass.

Manual verification

N/A — unit coverage drives the real routes and turn binding; no UI or live collector is involved.

Follow-ups

  • A worker turn still streaming when its item closes adds nothing to that item's roll-up: the outcome waits only for the acting (conductor) turn. Rare, since a conductor closes after the worker's done turn has ended. Fixing it means also waiting on the worker's open turn.
  • An item that never closes (its board purged) keeps a roll-up entry in the shared, capped map until the process restarts.
  • A goal's last round (every single-round goal included) gets no ultra outcome marker, and keeps its roll-up entry, until a later goal write moves the board on: the ledger has no "goal finished" action to end it on. The item outcomes and every turn are still in the trace. Ending it needs a ledger signal for a finished goal, or ending the round when its last item closes in a way a later create can reopen.
  • Continuations are counted per process: a restart on a round already past its six-day age budget writes its (cont 2) anchor again, with the same ids. Storing the live continuation on the header would stop that.
  • An item open across a restart, or created while tracing was off, has no roll-up, so its outcome carries no totals.
  • Phase 5b: worker creation spans, report → wake links, undelivered-report events, manager-change spans and patrol health counts.

Related Issues

Part of #1643

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@kyleseaman
kyleseaman requested review from a team and KiroCrewCaptain as code owners November 13, 2026 19:54
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 13, 2026
@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 48c3ad78094a9da82d186041719c5c1798bd6419 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 48c3ad78094a9da82d186041719c5c1798bd6419 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: this is tracing that is off by default, the ledger keys it adds only feed traces, and a round's trace ids are derived from the ledger itself, so nothing is held open across a restart.

Notes

  • goal_id and round_started_at are written to every board header even when tracing is off, and the PR lists that. They key only traces, so the cost is about 70 bytes per header, which is why test_an_oversized_acceptance_is_refused was raised from 360 to 440.
  • The module's own docstring lists a duplicate (cont 2) anchor after a restart. If duplicate anchors turn out to matter in Langfuse or X-Ray, fix it by storing the live continuation on the header, which phase 5b can do. Adding more in-memory registries would not fix it.

[DESIGN-REVIEWED] 48c3ad7

Description read

[DESCRIPTION-READ] 2ed0b5953528ddb633c506e8b2747bee4828d8863bdc05c4f837d06007df5c9b

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of 48c3ad78094a9da82d186041719c5c1798bd6419 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Every item maps to accepted RFC rfc-otel-tracing §5.5 and §10 row 5. That covers the new run default and the traces_ultra_mode: "turn" escape hatch. Each new surface has at least one real consumer.

What this change ships

Inventory (9 items) — 9 justified

Intent: with tracing on, an operator sees one Ultra Mode or Bot round as one waterfall trace (its turns, items, ledger writes and outcomes). This is an ADDITION.

  1. Under tracing (default on), a conductor or worker turn on a board now sits in its round's trace, not its own root trace — justified
  2. Each round gets an ultra round N anchor, linked to the previous round — justified
  3. Each item gets a work_item <id> anchor in its round, re-anchored when it is carried into the next round — justified
  4. Each ledger write (create, bind, verdict, close, report, steer, remote report) leaves a kirocrew.work.<action> span — justified
  5. work_item outcome and ultra outcome closed|advanced markers with summed spend, written after the acting turn ends — justified
  6. New config key telemetry.traces_ultra_mode (run/turn) — justified
  7. New ledger header keys goal_id and round_started_at, kept across a rebuild — justified
  8. Over-budget or six-day-old rounds roll over to an (cont 2) trace — justified
  9. runs.outcome takes an optional attrs=, and ten new allowlisted kirocrew.work.* attributes — justified

[FIRST-PRINCIPLES-REVIEWED] 48c3ad7

Description read

[DESCRIPTION-READ] 30628b5f7be5020fd9fb189e52f050d78f74433a9375670d2a6d6c0b7dd2f84f

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 48c3ad78094a9da82d186041719c5c1798bd6419 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 48c3ad7

Verdict parsed from the review's SHA-scoped output markers for commit 48c3ad78094a9da82d186041719c5c1798bd6419.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 48c3ad78094a9da82d186041719c5c1798bd6419 false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 48c3ad78094a9da82d186041719c5c1798bd6419 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 48c3ad7

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 48c3ad78094a9da82d186041719c5c1798bd6419 false-positive: <reason, 20 to 500 characters>

@github-actions github-actions Bot added the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Nov 13, 2026
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 13, 2026
@KiroCrewCaptain

KiroCrewCaptain commented Nov 13, 2026 •

Copy link
Copy Markdown
Collaborator

Code change approver recommendation: A person should decide

Why: the decider thinks the change is probably fine but estimates a 20% chance that this change introduces a defect, above the 12% limit for approving on its own.

This PR adds tracing of Ultra Mode and Bot rounds; tracing is off by default. Every new hook returns once provider.peek() reports tracing off. The ledger change is two header keys (goal_id, round_started_at), written under the existing lock and snapshot, so an undo restores them; a rebuild keeps them, and the 1 MB production record cap is unaffected. Round and item ids, and the roll-up totals, come out the same on the write routes and in turn binding, and the follow-up step after a turn is added under a lock, so it is neither lost nor run twice. Every route that writes to the ledger calls recorded only after its crew-log entry landed. Span attributes are allowlisted ids, fixed-vocabulary values and counts, run through the existing cleaning and redaction. The new tests drive the real routes and turn binding across round changes, rollover, restart, rebuild, turn mode, tracing off and a no-free-text check. The remaining gaps (anchors missing until the next goal write on old boards, thin tests for two routes) are known and non-blocking.

An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit 48c3ad7809 · 1421 changed lines. If this recommendation is wrong, reply here: replies are how the approver is evaluated.

@kyleseaman
kyleseaman force-pushed the crew/devmate/otel-pr5 branch from 6430b78 to 48c3ad7 Compare November 13, 2026 21:42
@kyleseaman

Copy link
Copy Markdown
Contributor Author

self-added: yes
mechanism: round start time persisted on the ledger header for R5's age budget

  • span=b66c053ac95c Preserve the round start time for age-based rollover

fixed in 48c3ad7: the ledger header carries round_started_at (stamped with the board, on a round change and on a new goal; a rebuild keeps it), and tracing/work.py parses it into the round and item RunRefs, so runs.over_budget applies the six-day age check after a restart too.
Test: test_a_round_older_than_the_age_budget_rolls_over_after_a_restart fails on 6430b78 and passes on 48c3ad7.

@kyleseaman

Copy link
Copy Markdown
Contributor Author

self-added: yes
mechanism: item roll-up keyed per item instead of per round anchor

  • span=ab8f39e2eeb3 An open item's spend is dropped when the round moves on

fixed in 48c3ad7: an item's roll-up is keyed by (conductor, generation, item id), opened once on create and closed only by the item's close; a round change no longer pops it, so work_item outcome sums every turn the item took.
Test: test_an_item_open_across_a_round_change_keeps_its_whole_spend fails on 6430b78 (1.0 of 3.0 credits) and passes on 48c3ad7.

@kyleseaman

Copy link
Copy Markdown
Contributor Author

Rebased: onto main ac4062d (from 0e5b203) to clear the conflict with #36658 in work_ledger_worker.py and to pick up #36804's coverage fix; head 6430b78 -> 48c3ad7, CI re-running.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention merge conflict Branch has merge conflicts with its base — author must resolve before merge readiness: checking Automated validation is still running labels Nov 13, 2026

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Read the whole diff (24 files, 1878 lines). This is RFC section 10 row 5 (phase 5a) of the tracing RFC that its owner accepted in full, the same RFC as phase 3a. The RFC's own text names every piece this PR adds: the round and item anchors and outcomes, goal_id on the ledger header, the ledger-write spans, and the traces_ultra_mode setting (Q9, default one trace per round, turn keeps per-turn traces).

Checked:

  • Off path. With telemetry.traces_enabled off (the default), header_before reads nothing, recorded returns on provider.peek() is None, and binding returns (None, ""). A test pins that nothing is read or emitted.
  • Data leaving the host. The new span attributes are ids and closed vocabularies (action, actor, state, verdict, status, reason, counts, a PR number). _write_attrs runs _enum over every value against work_vocab, so free text cannot pass, and a test feeds a sentinel to every free-text field and checks that no span carries it. No new export or destination.
  • The one change that runs with tracing off is the ledger header: goal_id and round_started_at are minted by the ledger. from_dict reads an older header without them, a rebuild keeps the cache's values, and the record cap test moves from 360 to 440 bytes only because the header grew. No crew-log entry changes.
  • No new refusal, and no regex in product code.
  • The RFC edits (outcome named ultra outcome <closed|advanced>, item anchor keyed by goal and round) record what the code does and stay inside the accepted design.

The shortcut comments (no anchor for a round that began while tracing was off, a goal's last round has no outcome) are stated in the code and the metrics doc. 97 checks pass, 0 red, 0 pending.

buluoray pushed a commit that referenced this pull request Nov 15, 2026
…er write (5a, #1643) (#36833)

Co-authored-by: Kyle Seaman <kseam@dev-dsk-kseam-1b-55230d27.us-east-1.amazon.com>
(cherry picked from commit 67014fc)
buluoray pushed a commit that referenced this pull request Nov 15, 2026
…er write (5a, #1643) (#36833)

Co-authored-by: Kyle Seaman <kseam@dev-dsk-kseam-1b-55230d27.us-east-1.amazon.com>
(cherry picked from commit 67014fc)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

readiness: passed Eligible automated validation passed for the current revision

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants