Repository navigation
feat(tracing): workflow, phase and outcome spans; completion joins the run (phase 3b, #1643) - #36717
Conversation
UX Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing blocks: the design reuses the 3a run-kind helper for workflows, adds no new UI, removes nothing, and its tracing hooks catch their own exceptions, so a tracing fault cannot fail a run. Notes
[DESIGN-REVIEWED] bc86ef1 Description read[DESCRIPTION-READ] 93c90bc28bfccf14d0a7edb0dd8b5fe5caa846a11a068b048f8a2782acaf686c That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Nothing blocks. Tracing is off by default, and the RFC status list from the base marks the tracing RFC Notes
What this change shipsInventory (8 items) — 7 justifiedIntent: with tracing on, an operator sees a workflow run as one trace (run, phases, calls, outcome, and the chat turn that reported the result). This is an ADDITION.
[FIRST-PRINCIPLES-REVIEWED] bc86ef1 Description read[DESCRIPTION-READ] 9e346c50caece80c63be154d9c30f64f8ca7dd563bace96b6d0ec06db08cb809 That sha256 names the PR description this verdict read. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
Code change approver recommendation: A person should decideWhy: the decider thinks the change is probably fine but estimates a 15% chance that this change introduces a defect, above the 12% limit for approving on its own. This PR adds tracing spans for workflow runs: a root span per run, a span per phase, each agent call under its phase, and an outcome marker. The chat turn that reports the result joins the run's trace through a one-shot hand-over, on the start-now, queued and retry paths. I checked the run/_run split against every caller, the bind/release pairing in ctx.agent (including cancels and replays), how phases close while parallel calls are still running, the single result-injection path, how the drain's context variable is scoped, the in-memory caps, the label redaction for script text, and the changed cron outcome attributes. The tests drive the real runner and the real bind_turn. I found no regression, fail-open path or unmet goal; the remaining gaps are small or acknowledged trace-attribution issues. An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit |
2044592 to
44f5dd3
Compare
|
Rebased: main moved in src/kiro_crew/dashboard/warm_task_brief.py, src/kiro_crew/workflows/runner.py and 99 other commits since this head went green (4d445d8 -> bd08975); CI re-running. The review lanes diffed the PR merge ref against a stale base (c4c95ce), so main-only changes such as #33805 (warm_task_brief.py) leaked into what they reviewed; the GPT block on warm_task_brief.py:1102 came from that. New head 44f5dd3; the PR diff itself is unchanged. |
…e run (#1643) Phase 3b of the native tracing RFC. A workflow run is now one trace, as one more run kind of the 3a surface root helper (TRIGGER_WORKFLOW): WorkflowRunner.run opens a "workflow <name>" anchor around the whole run and ends it with "workflow outcome <status>" and the spend roll-up. Each ctx.phase() is a "phase <title>" span from its phase_started event to the next phase or the run's end, with its own roll-up. A phase that a parallel branch closes while another branch's call under it still runs is exported once that call is released, so its roll-up keeps the call's spend. Each live ctx.agent() call is one turn under its phase, named "invoke_agent <phase>:<label>", with a turn id derived from the run and agent id, so running the same run again rebuilds the same tree. A call replayed from a prior run streams no turn, so a resumed run emits no span for it. run_background pins the turn that launched the run, and the completion turn the dashboard starts joins the run's trace under its root and links to that launcher, whether it starts at once or drains from the queue (meta.workflow_run_id). A retry of that turn carries the same meta, so it joins the same run. phase_started gains phase_index (the phase's 1-based ordinal), and agent_started gains the index of the phase its call ran in, so a consumer never recounts phases.
A run started from an intent authors its script in a task the runner starts for the author. A turn root bound by the runner's own task is refused there, so the authoring turns rooted unrelated traces, the Authoring phase had no model children and the outcome understated the run's spend. The author task now binds its own call under the Authoring phase and releases it when authoring ends. The two fixed sleeps in the workflow tracing tests are replaced with event barriers (await_until, until_parked), and the dashboard composition pins gain the workflow auto-turn's lazy tracing import and its extra_meta argument.
44f5dd3 to
bc86ef1
Compare
|
Rebased: main moved in src/kiro_crew/dashboard/trace_binding.py and docs/system-specs/modules/metrics.md since this head went green (bd08975 -> 8d481c3); CI re-running. #36833 (phase 5a, Ultra Mode rounds) changed bind_turn, and git merge-tree reported conflicts in both files. Resolved to keep both phases: a turn handed a workflow run (or a sub-agent run) joins that run; any other turn on a crew board joins its round with its rollup_key; the workflow run id is still merged after the crew read. New head bc86ef1. |
bolichen97
left a comment
There was a problem hiding this comment.
Read the whole diff (17 files, 1562 lines). This is RFC section 10 row 3b (workflows), the half of row 3 that phase 3a split off, under the tracing RFC its owner accepted in full. No new config key, export or destination.
Checked:
- Off path. With
telemetry.traces_enabledoff,runskips the script validation,observehands the event callback back untouched,bind_agentandpin_launcherreturn atprovider.peek() is None, and a test pins that no run is kept. - Data in spans. Names come from script text (workflow name, phase title,
label=). Each is collapsed to one line, passed whole throughvalidate_attrs, then cut to 64 characters. A call with nolabel=is named by its call id, never its prompt, and a test checks the prompt text stays out of every span name. A second test checks a credential-shaped title is redacted before the cut. The new attributes are the run id, the call id and the phase ordinal. - Behavior with tracing off. The one change visible then is additive event data:
phase_startedandagent_startedcarryphase_index, and a queued workflow completion prompt carriesmeta.workflow_run_id.phase_indexis not inREQUIRED_DATA_KEYS, so older journals still deserialize, and the consumers I checked on main (registry.py,preview.py) read onlytitle.enqueue_or_run_promptalready takesextra_meta. WorkflowRunner.runnow wraps the old body, renamed_run, with the same arguments and result. A failed or cancelled result sets the outcome status, and the open phase is ended in afinally.- No new refusal, and no regex in product code. 99 checks pass, 0 red, 0 pending.
The shortcut comments (runs and launchers kept in memory, so a completion turn after a restart roots its own trace) are stated in the code and the metrics doc.
…e run (phase 3b, #1643) (#36717) * feat(tracing): workflow, phase and outcome spans; completion joins the run (#1643) Phase 3b of the native tracing RFC. A workflow run is now one trace, as one more run kind of the 3a surface root helper (TRIGGER_WORKFLOW): WorkflowRunner.run opens a "workflow <name>" anchor around the whole run and ends it with "workflow outcome <status>" and the spend roll-up. Each ctx.phase() is a "phase <title>" span from its phase_started event to the next phase or the run's end, with its own roll-up. A phase that a parallel branch closes while another branch's call under it still runs is exported once that call is released, so its roll-up keeps the call's spend. Each live ctx.agent() call is one turn under its phase, named "invoke_agent <phase>:<label>", with a turn id derived from the run and agent id, so running the same run again rebuilds the same tree. A call replayed from a prior run streams no turn, so a resumed run emits no span for it. run_background pins the turn that launched the run, and the completion turn the dashboard starts joins the run's trace under its root and links to that launcher, whether it starts at once or drains from the queue (meta.workflow_run_id). A retry of that turn carries the same meta, so it joins the same run. phase_started gains phase_index (the phase's 1-based ordinal), and agent_started gains the index of the phase its call ran in, so a consumer never recounts phases. * fix(tracing): trace a workflow's authoring turn under its phase A run started from an intent authors its script in a task the runner starts for the author. A turn root bound by the runner's own task is refused there, so the authoring turns rooted unrelated traces, the Authoring phase had no model children and the outcome understated the run's spend. The author task now binds its own call under the Authoring phase and releases it when authoring ends. The two fixed sleeps in the workflow tracing tests are replaced with event barriers (await_until, until_parked), and the dashboard composition pins gain the workflow auto-turn's lazy tracing import and its extra_meta argument. --------- Co-authored-by: Kyle Seaman <kseam@dev-dsk-kseam-1b-55230d27.us-east-1.amazon.com> (cherry picked from commit 875b273)
…e run (phase 3b, #1643) (#36717) * feat(tracing): workflow, phase and outcome spans; completion joins the run (#1643) Phase 3b of the native tracing RFC. A workflow run is now one trace, as one more run kind of the 3a surface root helper (TRIGGER_WORKFLOW): WorkflowRunner.run opens a "workflow <name>" anchor around the whole run and ends it with "workflow outcome <status>" and the spend roll-up. Each ctx.phase() is a "phase <title>" span from its phase_started event to the next phase or the run's end, with its own roll-up. A phase that a parallel branch closes while another branch's call under it still runs is exported once that call is released, so its roll-up keeps the call's spend. Each live ctx.agent() call is one turn under its phase, named "invoke_agent <phase>:<label>", with a turn id derived from the run and agent id, so running the same run again rebuilds the same tree. A call replayed from a prior run streams no turn, so a resumed run emits no span for it. run_background pins the turn that launched the run, and the completion turn the dashboard starts joins the run's trace under its root and links to that launcher, whether it starts at once or drains from the queue (meta.workflow_run_id). A retry of that turn carries the same meta, so it joins the same run. phase_started gains phase_index (the phase's 1-based ordinal), and agent_started gains the index of the phase its call ran in, so a consumer never recounts phases. * fix(tracing): trace a workflow's authoring turn under its phase A run started from an intent authors its script in a task the runner starts for the author. A turn root bound by the runner's own task is refused there, so the authoring turns rooted unrelated traces, the Authoring phase had no model children and the outcome understated the run's spend. The author task now binds its own call under the Authoring phase and releases it when authoring ends. The two fixed sleeps in the workflow tracing tests are replaced with event barriers (await_until, until_parked), and the dashboard composition pins gain the workflow auto-turn's lazy tracing import and its extra_meta argument. --------- Co-authored-by: Kyle Seaman <kseam@dev-dsk-kseam-1b-55230d27.us-east-1.amazon.com> (cherry picked from commit 875b273)
Problem / Motivation
Goal: With tracing on, a workflow run shows up as one trace: the run, each of its phases, each agent call under its phase, how the run ended, and the chat turn that reported its result.
Today a workflow run has no root span. Each
ctx.agent()step's turn mints its own random trace, so a ten-step run is ten unrelated traces with nothing saying which run, phase or call they belong to. The turn that posts the run's result back to chat is unrelated to the run as well.Why it matters
The tracing RFC's goal G3 is that the tree of who-started-what (workflows included) can be rebuilt from parent spans alone. Workflows are where many model calls fan out in parallel. Without one trace per run, an operator cannot see which phase a slow or expensive call was in, what the run cost in total, or which chat turn launched it and which one answered.
Not a goal
What changed (motivation → approach → change)
Design record: the native tracing RFC §5.6 (span names), §5.7 (workflows), §5.8 (ids), §10 row 3. 3a (#36153) gave each background engine a root span through one helper,
tracing/surfaces.py; this is 3b.Terms: a run kind is an engine whose run is one trace, with an anchor (the root span) and an outcome (a marker under it saying how the run ended). A roll-up is the summed spend (turns, tokens, credits) of the model turns under a span.
Approach: a workflow run is one more run kind of that helper (
TRIGGER_WORKFLOW); the workflow-only parts live in a newtracing/workflows.py.workflows/runner.py::WorkflowRunner.runopens aworkflow <name>anchor (keyed byrun_id, named byMETA["name"]) around the whole run, and ends it withworkflow outcome <status>and the run's roll-up. The old body is now_run, unchanged;runforwards**kwargs. With tracing on,runre-parses the script for its name.phase_startedopens aphase <title>span under the anchor with its calls' roll-up. With no phase-end event, it ends where the next phase starts or the run ends. If a parallel branch starts the next phase while a call under this one still runs, the span is exported when that call finishes, still ending at the boundary, so the roll-up keeps that call.ctx.agent()binds each live call as a turn under its phase (bind_agent) in the task that streams it, so eachctx.parallelbranch gets its own; a pooled step'sasyncio.wait_forstays in that task on Python 3.12+. Namedinvoke_agent <phase>:<label>, or<phase>:a<call_index>withoutlabel=(the fallback label is prompt text). A call outside any phase hangs under the anchor. The turn id derives from(run_id, agent_id), so the same run rebuilds the same tree. A call replayed on resume streams no turn, so it emits no span.run_backgroundrecords the launching session's open turn (turns.open_turn_of_session).workflow_startup.py's auto-turn marks the run on the turn it starts (delivering); a queued turn's entry carriesmeta.workflow_run_id, whichqueue_drain.pyreads back, and a retry (_queue_recovery) carries the same meta.trace_binding.bind_turntakes the run once, joins its trace under the anchor (RFC R4: a turn whose one cause is a finished run joins it), links to the launcher and carrieskirocrew.workflow_run_id. Sub-agent links on the same drain are kept; a lone sub-agent's join gives way. Runs and launchers are kept in memory (last 1,024), so after a restart the turn roots its own trace.workflows/events.py). One new key,phase_index(1-based), onphase_started, and on everyagent_started(0 outside a phase). Not required, so old journals load.kirocrew.workflow_run_id,kirocrew.agent_id,kirocrew.phase_index. Names, titles and labels are script text: collapsed to one line, passed whole throughvalidate_attrs, then cut to 64 characters.workflows.md(events, Traces) and metrics.md Traces, in this commit.Rejected: a second, workflow-specific root implementation, and rebuilding call spans from
agent_started/agent_finished, which lack the turn'schatand tool spans.Backwards compatibility
Compatible: tracing is off by default. Off,
runadds the helper's switch read and one context-variable write, a table lookup at the end, and passes events through untouched; the auto-turn and the drain each set one context variable.phase_indexis an extra optional event key, andvalidate_eventallows extra keys.WorkflowRunner.runaccepts the same keywords.Changed for tracing users: every run kind's outcome marker now carries its anchor's keys (
runs.outcome(attrs=)), so cron outcomes gain them too.Removes nothing: the diff adds spans, one optional event key and three allowlisted attributes;
run's body moves unchanged into_run.Tests
test/tracing/test_tracing_workflows.py(new, 17 tests), driving the realWorkflowRunner.runwith a stub step that streams a traced turn:session.id, the launching session's.asyncio.wait_forkeeps its turn under its phase.Authoringphase (the author runs in its own task, so it binds its own call there), and its spend rolls up to the phase and the run.run_idagain gives the identical span ids and parents, and both runs pin the shape (anchor → three phases in order → their agent turns → outcome with roll-up).invoke_agentorchatspan, and nothing in the first run's trace.workflow outcome failed; a cancelled one endsworkflow outcome cancelledand still ends its phase.phase_indexridesphase_startedandagent_started, tracing on or off; off, no run is kept.bind_turnin a task started insidedelivering, carries the run'strace_id, sits under the anchor, keeps its ownturn_id, and links to the launching turn.run_of, taken once.test/tracing/test_tracing_completion_retry.pygains a workflow case: a completion turn whose runtime dies is retried through the real_queue_recovery, and both attempts join the run.Failing first: with the runner, event and
bind_turnwiring removed, 9 of the first 10 tests fail (the tenth tests the one-shot hand-over on its own). With onlybind_turnremoved the completion test fails, and with the base runner the replay shape test fails. The content-safety tests, the completion turn's run id, the in-flight phase roll-up and the retry case each failed before their fixes. The authoring test failed before the second commit. The pooled-step, cancelled and off tests pin behavior that already held. The tests wait on events, never fixed sleeps.test/tracing/test_tracing_ids_attrs.py::test_the_attribute_allowlist_is_pinnedgains the three keys.Manual verification
N/A for manual steps: the tests drive the real runner and the real dashboard binding with an in-memory exporter; no UI changes.
Local pre-push evidence:
test/tracing(275 tests) plus every workflow test file pass; mypy, flake8, isort, black, docs-lint, comment-history, harness parity and the line cap pass. The local-gate backend batches (all 6) failed only in tests that fail the same way on base4d445d8a76: AI-review workflow, file explorer, design-tweak, CLI doctor/STT, terminal, macOS release and a few single tests.test_auto_improvement_members(CI shard 7) also fails on base.Related Issues
Part of #1643 (phase 3b; follows #36153).
Follow-ups
run_idplumbed fromWorkflowService.rerun_subtreeinto the runner.otlpextra, so the wire-level tracing tests skip in CI (carried from feat(tracing): OTLP span export to Langfuse and CloudWatch (phase 2, #1643) #35792).Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)Contribution License Agreement
Contributions are accepted under the project's Apache-2.0 license.