Repository navigation
feat(tracing): surface roots for channels, cron, heartbeat, background, webhooks and the task runner (phase 3a, #1643) - #36153
Conversation
UX Review (Opus 5.5) — ⏭️ skippedRevision |
First Principles Review (Opus 5.5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Nothing blocks: every root, the cron anchor and its outcome map to RFC row 3 ( Notes
What this change shipsInventory (9 items) — 7 justifiedIntent: with tracing on, an operator can see what started each non-dashboard turn, and a cron run, retries included, shows up as one trace. This is an ADDITION.
[FIRST-PRINCIPLES-REVIEWED] e4c160a Description read[DESCRIPTION-READ] 74a623a91dc792fb9957fdcd3fe992914fe54e2efcb1e860d5685986f914922e That sha256 names the PR description this verdict read. Recompute it with |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing blocks: one Notes
[DESIGN-REVIEWED] e4c160a Description read[DESCRIPTION-READ] 772e7567420deb5f5556010155ad37cf4771f0f7f4187074ad0699cd454d0002 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
626173d to
e72b154
Compare
Code change approver recommendation: A person should decideWhy: the decider thinks the change is probably fine but estimates a 14% chance that this change introduces a defect, above the 12% limit for approving on its own. The PR adds one surfaces.root/rooted helper and wires it into the channel driver, heartbeat, background, webhook, task-runner and cron engines; a cron run becomes one trace (an anchor, its turns as children, an outcome span), with retries as attempts of the same turn. Checked in the code: on Python ≥3.12, wait_for runs in the current task, so the cron root is still bound when the callback's turns stream. adopt_root and next_turn behave correctly with fair admission on and off, and the dispatch is closed when admission never starts it. No dashboard or sub-agent root is overridden: _run_chat never calls TurnDriver.run, and no cron or sub-agent path calls run_bg_oneliner. Every new attribute key is allowlisted, channel values come from a closed vocabulary, and no job name reaches a span. With tracing off, a root is one context-variable write. Tests drive each engine's real entry point, and CI is green. The remaining points are documentation alignment and future-proofing, not defects. An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit |
e72b154 to
7ff1b31
Compare
Phase 3a of the native tracing RFC. One helper, tracing/surfaces.py, opens the root every engine that starts a turn outside _run_chat now uses: channels (TurnDriver.run), heartbeat tasks, background one-liners and background turns, webhooks, task-runner steps, and cron runs. Every span of those turns carries kirocrew.trigger, plus kirocrew.channel on a channel turn. Every prompt inside one root is the next attempt of the same turn, so a retry is attempt 2, not a new trace. Cron is a run kind: a "cron <job kind>" anchor keyed (job id, marker_run), the run's turns under it, and a "cron outcome <status>" marker with the spend roll-up. A script or command run adds a "cron run <kind>" step span and has no chat span. Off, a root is one context-variable write and imports no SDK. The workflow root kind (phase 3b) follows in its own change.
|
Rebased onto Why: Backend Tests (3.12, 8) and (Windows) (8) failed on The rebase was clean, with no conflicts and no change to the diff of this PR. On the rebased tree, |
7ff1b31 to
e4c160a
Compare
|
Linked #1643 is a T3 item still waiting on a maintainer design sign-off, so this cannot merge until a maintainer records it. A maintainer will post the decision here. Review result on this head: no blocking findings. The review followed the cron path through Optional:
|
bolichen97
left a comment
There was a problem hiding this comment.
Approve. I read the whole diff on this head, tests included. All checks are green.
Scope. This is RFC §10 row 3, split into 3a (this PR) and 3b (workflows). It is the next phase of the design the RFC's owner accepted, and it lands PR by PR like 1a, 1b and 2. It is about 650 lines of product code and docs, well inside row 3's ~950 budget, and the rest is tests.
Data leaving the host. There is no new export, destination, config key or setting. Everything stays behind the existing telemetry.traces_enabled switch, which defaults to off, and leaves only through the phase 2 OTLP exporter when a collector is named. The new attributes are a closed vocabulary added to ALLOWED_KEYS:
kirocrew.trigger(six constants, and an unknown value raises).kirocrew.channel(only aCHANNEL_SESSION_NAMESPACESvalue, never a renderer label outside it).kirocrew.cron.job_kind(script, command or agent).kirocrew.cron.job_id, raw like the other ids. The anchor is named by job kind, never the job name, and a test checks the name stays out of every span.
Off path. With tracing off, surfaces.root is one context-variable write. test_off_a_root_imports_no_sdk checks that OpenTelemetry is never imported.
Behavior. _open_run and _close_run catch every error and log it at DEBUG, so tracing cannot fail a cron run. TurnDriver.run only splits into run and _drive. The cron body under _execute_with_timeout is the same code, indented under the root. admitted_cron_dispatch now wraps the dispatch so the admitted task adopts the run's root, and it closes a dispatch that never started instead of leaving it un-awaited. Nothing new is refused, and there is no regex.
Phase 3a direction signed by maintainer (bolichen97) as the RFC's row 3. Phase 3b (workflow roots) gets its own review.
Problem / Motivation
Goal: With tracing on, a turn started by a channel message, a cron run, a heartbeat, a background job, a webhook or a task-runner step shows up as a trace that says what started it, and a cron run is one trace with its retries inside it.
Today only dashboard turns and sub-agent runs have a root. Every other engine's turn mints a random turn id per stream: nothing on the span says it came from Slack or a cron job, a retried cron turn becomes a second unrelated trace, and a script or command cron run, which streams no model turn, leaves no span at all.
Why it matters
The tracing RFC's first goal (G1) is that every model turn on every surface produces a trace with one tree shape. Unattended engines (cron, heartbeat, webhooks, the task runner) are where an operator most needs to see what ran, what it cost and where it failed, and they are the ones with no trigger today.
Not a goal
workflow/phase/workflow outcomespans,phase_index, the workflow completion turn and resume. That is phase 3b, which follows this PR (RFC §10 row 3 was split in two).dashboard/trace_binding.py) or to sub-agent roots.What changed (motivation → approach → change)
Design record: the native tracing RFC, §5.2 layer 2 and §10 row 3. #1643 ships in phases: 1a/1b (turn spans, sub-agent and run roots) and 2 (OTLP export) are merged; 3 is split into this PR and 3b (workflows).
Approach: one helper for every engine, so every root kind gets the same span tree and 3b's workflow root is one more kind, not a special case.
tracing/surfaces.py(new).root(trigger, ...)binds a turn root for the block withkirocrew.trigger(closed vocabulary:channel,cron,heartbeat,background,webhook,task) and, for a channel,kirocrew.channel. Every prompt inside one block is the nextkirocrew.attemptof the same turn, so a retry is attempt 2, not a new trace.rooted(trigger)is the decorator form. Tracing failures degrade to self-rooted turns; only an unknown trigger raises.tracing/runs.py: a zero-lengthcron <job kind>anchor keyed by job id andmarker_run(the claim's per-run token), the run's turns as its children, andcron outcome <status>with their summed spend. A script or command job adds acron run <kind>span and has nochatspan. Status:failed(callback error, deadline, or reaper kill),cancelled(user or shutdown),skipped(run_never_started), elsefinished. Root sessioncron:<job id>, so onesession.idper trace.messaging/driver.py::TurnDriver.run(body moved unchanged to_drive; the channel comes from the renderer's transport, so a dashboard-linked Slack reply staysslack);slack/gateway_runtime/heartbeat_turns.py;llm_helpers_parts/background_turns.py(run_bg_oneliner,background_turn);dashboard/handlers/hooks.py::_run_hook_inner;task_executor.py(execute_task,self_review);cron_service/service/run_execution.py::_execute_with_timeout, around the deadline. The gateway's_cron_callback(unchanged) retries a transient error by calling itself in the same task, which is why a retry is the next attempt; each agent of anagent_sequencestarts its own turn of the run (turns.next_turn, inslack/gateway_runtime/cron_turn.py).shared_crew/admission_entry.py::admitted_cron_dispatchruns the dispatch in a new task. A root ignores turns from tasks other than the one that bound it, so the dispatch first takes it over (turns.adopt_root); a dispatch admission never starts is now closed, not left un-awaited.kirocrew.cron.job_id,kirocrew.trigger,kirocrew.channel,kirocrew.cron.job_kind. No job name, prompt or job text reaches a span.metrics.mdTraces and therun_execution.pyrow inlearn-cron-dashboard.md.src/kiro_crew/llm_helpers.pynames the newtrace_surfacesowner global for type checkers, andtest/test_transport_permission_floor.pyfollows the approval calls fromTurnDriver.runinto_drive(moved unchanged).Rejected: a per-engine root (3b would add a seventh shape) and a new trace per cron retry (the retry is the same run).
Backwards compatibility
Compatible: tracing is off by default, and off a root is one context-variable write with no SDK import. With tracing on, spans only gain attributes and a parent; no key, config field or public function is removed or renamed.
Removes nothing: the diff only adds a helper, four allowlisted keys and roots around existing calls;
TurnDriver.runkeeps its signature andjob_kindreturns the same three strings the inline branch did.Tests
test/tracing/test_tracing_surfaces.py(new, 26 tests):chat; a raising run isfailed/cancelled; a status set in the block lands on the outcome; an unknown trigger is refused;rootedgives each call its own root; off, a root imports no OpenTelemetry (subprocess).session.id, and no span carries the job name; a retried cron run is attempt 2 under the same anchor withturns_total == 2; a script cron run has acron run scriptspan and nochatspan; a run its deadline ends, or the reaper kills, isfailed, notcancelled; a run whose payload never started isskipped; a channel turn carrieskirocrew.channel=telegram; a background one-liner, a background turn, a webhook turn, a task-runner step, a task-runner self-review and a heartbeat task each root with their trigger._cron_callbackwith a two-agent sequence: two turns, attempt 1 each, both under the anchor (turns_total == 2).test/test_shared_crew_admission_entry.py::test_an_admitted_cron_dispatch_streams_under_its_runs_trace_root: with fair admission on, the dispatch runs in its own task and its turn still joins the run's anchor (turns.adopt_root).test/test_shared_crew_admission_entry.py::test_a_gateway_cron_run_under_fair_admission_joins_its_anchor: the gateway's real_cron_callback, run byCronService._execute_with_timeoutwith fair admission on; it asserts the turn streamed admitted, in a task other than the run's, and still sits under thecron agentanchor with the roll-up. Both admission tests fail withadopt_rootremoved.test/tracing/test_tracing_ids_attrs.py::test_the_attribute_allowlist_is_pinnedupdated with the four keys.Failing first: with the six engine files stashed, the 9 engine tests written in the first round fail (no anchor, missing
kirocrew.trigger). Each review-round fix was shown the same way: without it, the deadline,session.id, never-started, reaper, roll-up and admission tests fail (7 in all, with the self-review root). With everything, the 26 surface tests and the admission-entry file pass.Manual verification
N/A for manual steps: unit coverage drives each engine's real entry point with an in-memory exporter; no UI or external service changes.
Local pre-push evidence, and what did not complete:
claude-opus-5.5): three rounds; rounds 1 and 2 blocked, each fix landed with a failing-first test, round 3 found no blocking issues.gpt-6.1-sol): rate-limited by the model provider on all four tries, so no local GPT verdict exists. CI's GPT lane is the GPT review for this PR.d9839eee5d, in files this PR does not touch). Batches 2-4 did not complete locally (host load ~27) and were stopped; CI's full suite covers them.Related Issues
Part of #1643 (phase 3a). Phase 3b (workflow root kind) follows.
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)Contribution License Agreement
Contributions are accepted under the project's Apache-2.0 license.