Skip to content

feat(tracing): surface roots for channels, cron, heartbeat, background, webhooks and the task runner (phase 3a, #1643) - #36153

Merged
bolichen97 merged 1 commit into
mainfrom
crew/devmate/otel-pr3
Nov 13, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
crew/devmate/otel-pr3

Conversation

@kyleseaman

@kyleseaman kyleseaman commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: With tracing on, a turn started by a channel message, a cron run, a heartbeat, a background job, a webhook or a task-runner step shows up as a trace that says what started it, and a cron run is one trace with its retries inside it.

Today only dashboard turns and sub-agent runs have a root. Every other engine's turn mints a random turn id per stream: nothing on the span says it came from Slack or a cron job, a retried cron turn becomes a second unrelated trace, and a script or command cron run, which streams no model turn, leaves no span at all.

Why it matters

The tracing RFC's first goal (G1) is that every model turn on every surface produces a trace with one tree shape. Unattended engines (cron, heartbeat, webhooks, the task runner) are where an operator most needs to see what ran, what it cost and where it failed, and they are the ones with no trigger today.

Not a goal

  • The workflow root kind: workflow / phase / workflow outcome spans, phase_index, the workflow completion turn and resume. That is phase 3b, which follows this PR (RFC §10 row 3 was split in two).
  • Any change to the dashboard runner's own root (dashboard/trace_binding.py) or to sub-agent roots.
  • New export destinations, config keys or settings UI.
  • Spans for side chat, threads, apps, standalone compaction or reset.

What changed (motivation → approach → change)

Design record: the native tracing RFC, §5.2 layer 2 and §10 row 3. #1643 ships in phases: 1a/1b (turn spans, sub-agent and run roots) and 2 (OTLP export) are merged; 3 is split into this PR and 3b (workflows).

Approach: one helper for every engine, so every root kind gets the same span tree and 3b's workflow root is one more kind, not a special case.

  • tracing/surfaces.py (new). root(trigger, ...) binds a turn root for the block with kirocrew.trigger (closed vocabulary: channel, cron, heartbeat, background, webhook, task) and, for a channel, kirocrew.channel. Every prompt inside one block is the next kirocrew.attempt of the same turn, so a retry is attempt 2, not a new trace. rooted(trigger) is the decorator form. Tracing failures degrade to self-rooted turns; only an unknown trigger raises.
  • Cron is a run. Using phase 1b's tracing/runs.py: a zero-length cron <job kind> anchor keyed by job id and marker_run (the claim's per-run token), the run's turns as its children, and cron outcome <status> with their summed spend. A script or command job adds a cron run <kind> span and has no chat span. Status: failed (callback error, deadline, or reaper kill), cancelled (user or shutdown), skipped (run_never_started), else finished. Root session cron:<job id>, so one session.id per trace.
  • Engines wired: messaging/driver.py::TurnDriver.run (body moved unchanged to _drive; the channel comes from the renderer's transport, so a dashboard-linked Slack reply stays slack); slack/gateway_runtime/heartbeat_turns.py; llm_helpers_parts/background_turns.py (run_bg_oneliner, background_turn); dashboard/handlers/hooks.py::_run_hook_inner; task_executor.py (execute_task, self_review); cron_service/service/run_execution.py::_execute_with_timeout, around the deadline. The gateway's _cron_callback (unchanged) retries a transient error by calling itself in the same task, which is why a retry is the next attempt; each agent of an agent_sequence starts its own turn of the run (turns.next_turn, in slack/gateway_runtime/cron_turn.py).
  • Fair admission. With Shared Crew's per-principal gate on, shared_crew/admission_entry.py::admitted_cron_dispatch runs the dispatch in a new task. A root ignores turns from tasks other than the one that bound it, so the dispatch first takes it over (turns.adopt_root); a dispatch admission never starts is now closed, not left un-awaited.
  • Allowlist: kirocrew.cron.job_id, kirocrew.trigger, kirocrew.channel, kirocrew.cron.job_kind. No job name, prompt or job text reaches a span.
  • Nesting: a background one-liner a dashboard turn awaits takes over for its own turns; the dashboard root returns when it closes.
  • Specs: metrics.md Traces and the run_execution.py row in learn-cron-dashboard.md.
  • Pins: src/kiro_crew/llm_helpers.py names the new trace_surfaces owner global for type checkers, and test/test_transport_permission_floor.py follows the approval calls from TurnDriver.run into _drive (moved unchanged).

Rejected: a per-engine root (3b would add a seventh shape) and a new trace per cron retry (the retry is the same run).

Backwards compatibility

Compatible: tracing is off by default, and off a root is one context-variable write with no SDK import. With tracing on, spans only gain attributes and a parent; no key, config field or public function is removed or renamed.

Removes nothing: the diff only adds a helper, four allowlisted keys and roots around existing calls; TurnDriver.run keeps its signature and job_kind returns the same three strings the inline branch did.

Tests

test/tracing/test_tracing_surfaces.py (new, 26 tests):

  • Helper: a surface turn roots itself with its trigger and channel; prompts inside one root are attempts 1, 2 of one trace; a turn after the root closes is not part of it; an inner root is its own turn and the outer binding resumes; a run root anchors its turns with deterministic ids and rolls up their spend; a step run has a step span and no chat; a raising run is failed / cancelled; a status set in the block lands on the outcome; an unknown trigger is refused; rooted gives each call its own root; off, a root imports no OpenTelemetry (subprocess).
  • Engines (the RFC exit criteria): an agent cron run is one anchored trace, every span wears the job's session.id, and no span carries the job name; a retried cron run is attempt 2 under the same anchor with turns_total == 2; a script cron run has a cron run script span and no chat span; a run its deadline ends, or the reaper kills, is failed, not cancelled; a run whose payload never started is skipped; a channel turn carries kirocrew.channel=telegram; a background one-liner, a background turn, a webhook turn, a task-runner step, a task-runner self-review and a heartbeat task each root with their trigger.
  • A run whose step span cannot start leaves no roll-up open.
  • The gateway's real _cron_callback with a two-agent sequence: two turns, attempt 1 each, both under the anchor (turns_total == 2).
  • test/test_shared_crew_admission_entry.py::test_an_admitted_cron_dispatch_streams_under_its_runs_trace_root: with fair admission on, the dispatch runs in its own task and its turn still joins the run's anchor (turns.adopt_root).
  • test/test_shared_crew_admission_entry.py::test_a_gateway_cron_run_under_fair_admission_joins_its_anchor: the gateway's real _cron_callback, run by CronService._execute_with_timeout with fair admission on; it asserts the turn streamed admitted, in a task other than the run's, and still sits under the cron agent anchor with the roll-up. Both admission tests fail with adopt_root removed.
  • test/tracing/test_tracing_ids_attrs.py::test_the_attribute_allowlist_is_pinned updated with the four keys.

Failing first: with the six engine files stashed, the 9 engine tests written in the first round fail (no anchor, missing kirocrew.trigger). Each review-round fix was shown the same way: without it, the deadline, session.id, never-started, reaper, roll-up and admission tests fail (7 in all, with the self-review root). With everything, the 26 surface tests and the admission-entry file pass.

Manual verification

N/A for manual steps: unit coverage drives each engine's real entry point with an in-memory exporter; no UI or external service changes.

Local pre-push evidence, and what did not complete:

  • Opus local review (claude-opus-5.5): three rounds; rounds 1 and 2 blocked, each fix landed with a failing-first test, round 3 found no blocking issues.
  • GPT local review (gpt-6.1-sol): rate-limited by the model provider on all four tries, so no local GPT verdict exists. CI's GPT lane is the GPT review for this PR.
  • Local gate related tests: batch 1 of 4 ran (264 failures, the same set that fails on base d9839eee5d, in files this PR does not touch). Batches 2-4 did not complete locally (host load ~27) and were stopped; CI's full suite covers them.
  • Passed locally: mypy, flake8, isort, black, docs-lint, comment-history, file line cap, semgrep, and the tracing, cron, webhook, heartbeat, task-runner, driver, background-turn and admission-entry test files.

Related Issues

Part of #1643 (phase 3a). Phase 3b (workflow root kind) follows.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributions are accepted under the project's Apache-2.0 license.

@kyleseaman
kyleseaman requested review from a team and KiroCrewCaptain as code owners November 12, 2026 22:49
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 12, 2026
@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision e4c160a67ab36191ba11ca2241eeba2f8d3039ca touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of e4c160a67ab36191ba11ca2241eeba2f8d3039ca — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Nothing blocks: every root, the cron anchor and its outcome map to RFC row 3 (in-progress on base), and tracing stays off by default.

Notes

  • Item 7 — one consumer, generalized: RUN_KINDS = frozenset({TRIGGER_CRON}) and root()'s run_name/run_key/root_session_key/step/attrs parameters have exactly one caller, _run_root in run_execution.py. The description's own reason is a future one: "3b's workflow root is one more kind". Subtraction: until 3b lands, replace the set with trigger == TRIGGER_CRON.
  • Item 8 — duplicate of tracing/turns.py:655: the new open_rollup re-spells the inline if ... len(_rollups) < _MAX_OPEN: _rollups[...] = runs.Rollup() at the spawn site. Grepping _rollups\[ finds 2 writers. Subtraction: have the spawn site call open_rollup and delete the inline copy.

What this change ships

Inventory (9 items) — 7 justified

Intent: with tracing on, an operator can see what started each non-dashboard turn, and a cron run, retries included, shows up as one trace. This is an ADDITION.

  1. Channel-message turns carry kirocrew.trigger=channel and the transport's kirocrew.channel — justified
  2. Heartbeat, background, webhook and task-runner (step and self-review) turns each root with their trigger — justified
  3. Prompts inside one root are attempts of one turn, so a retry no longer starts a new trace — justified
  4. A cron run is one trace: a cron <kind> anchor, its turns under it, and a cron outcome <status> span with the summed spend — justified
  5. Script and command cron runs get a cron run <kind> span — justified
  6. Each agent of a cron agent_sequence is its own turn of the run (next_turn) — justified
  7. A generalized run-kind set and run parameters on root() — one consumer, generalized
  8. New open_rollup/close_rollup helpers — duplicate of the inline open at turns.py:655
  9. Under fair admission the cron dispatch takes over the root (adopt_root), and a dispatch that never starts is now closed — justified

[FIRST-PRINCIPLES-REVIEWED] e4c160a

Description read

[DESCRIPTION-READ] 74a623a91dc792fb9957fdcd3fe992914fe54e2efcb1e860d5685986f914922e

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of e4c160a67ab36191ba11ca2241eeba2f8d3039ca — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: one root() helper adds attributes and parents to spans, tracing stays off by default, and nothing is taken away.

Notes

  • When a dashboard turn awaits run_bg_oneliner, the one-liner's turns now get their own background root instead of joining the dashboard turn's trace. That changes how existing dashboard traces look, so check this split is the RFC's intended G1 tree before 3b builds on it.
  • adopt_root() hands the root to whichever task calls it, without checking anything. It is safe only because the binding task "streams nothing of its own after that". If 3b or another engine reuses it in a task that keeps streaming, both tasks will count turns on one root.

[DESIGN-REVIEWED] e4c160a

Description read

[DESCRIPTION-READ] 772e7567420deb5f5556010155ad37cf4771f0f7f4187074ad0699cd454d0002

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed e4c160a67ab36191ba11ca2241eeba2f8d3039ca — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] e4c160a

Verdict parsed from the review's SHA-scoped output markers for commit e4c160a67ab36191ba11ca2241eeba2f8d3039ca.

False positive or not applicable? A repository writer can comment:
/ai-review override fable e4c160a67ab36191ba11ca2241eeba2f8d3039ca false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of e4c160a67ab36191ba11ca2241eeba2f8d3039ca and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] e4c160a

False positive or not applicable? A repository writer can comment:
/ai-review override gpt e4c160a67ab36191ba11ca2241eeba2f8d3039ca false-positive: <reason, 20 to 500 characters>

@kyleseaman
kyleseaman force-pushed the crew/devmate/otel-pr3 branch 2 times, most recently from 626173d to e72b154 Compare November 12, 2026 23:59
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running and removed readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention labels Nov 13, 2026
@KiroCrewCaptain

KiroCrewCaptain commented Nov 13, 2026 •

Copy link
Copy Markdown
Collaborator

Code change approver recommendation: A person should decide

Why: the decider thinks the change is probably fine but estimates a 14% chance that this change introduces a defect, above the 12% limit for approving on its own.

The PR adds one surfaces.root/rooted helper and wires it into the channel driver, heartbeat, background, webhook, task-runner and cron engines; a cron run becomes one trace (an anchor, its turns as children, an outcome span), with retries as attempts of the same turn. Checked in the code: on Python ≥3.12, wait_for runs in the current task, so the cron root is still bound when the callback's turns stream. adopt_root and next_turn behave correctly with fair admission on and off, and the dispatch is closed when admission never starts it. No dashboard or sub-agent root is overridden: _run_chat never calls TurnDriver.run, and no cron or sub-agent path calls run_bg_oneliner. Every new attribute key is allowlisted, channel values come from a closed vocabulary, and no job name reaches a span. With tracing off, a root is one context-variable write. Tests drive each engine's real entry point, and CI is green. The remaining points are documentation alignment and future-proofing, not defects.

An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit e4c160a67a · 1259 changed lines. If this recommendation is wrong, reply here: replies are how the approver is evaluated.

@kyleseaman
kyleseaman force-pushed the crew/devmate/otel-pr3 branch from e72b154 to 7ff1b31 Compare November 13, 2026 01:39
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 13, 2026
Phase 3a of the native tracing RFC. One helper, tracing/surfaces.py,
opens the root every engine that starts a turn outside _run_chat now
uses: channels (TurnDriver.run), heartbeat tasks, background one-liners
and background turns, webhooks, task-runner steps, and cron runs. Every
span of those turns carries kirocrew.trigger, plus kirocrew.channel on
a channel turn.

Every prompt inside one root is the next attempt of the same turn, so a
retry is attempt 2, not a new trace. Cron is a run kind: a
"cron <job kind>" anchor keyed (job id, marker_run), the run's turns
under it, and a "cron outcome <status>" marker with the spend roll-up.
A script or command run adds a "cron run <kind>" step span and has no
chat span. Off, a root is one context-variable write and imports no
SDK.

The workflow root kind (phase 3b) follows in its own change.
@kyleseaman

kyleseaman commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor Author

Rebased onto main at b7a4e1e70c550bddfc0939bf2477310bbfcf024f: head 7ff1b31ada7f → e4c160a67ab3.

Why: Backend Tests (3.12, 8) and (Windows) (8) failed on test/test_reply_format_once.py::TestEveryMessageWhereTheBlockCannotBeTakenOut::test_the_table_names_every_selectable_harness (mimocode missing from its table). That is a red on main, tracked in #36279 and fixed by #36290, which merged after the merge commit for this PR was built. A re-run reuses the old merge commit, so it failed the same way again.

The rebase was clean, with no conflicts and no change to the diff of this PR. On the rebased tree, test/tracing and the suites this PR touches pass locally (1,384 tests, test_reply_format_once.py included), as do mypy, flake8, docs-lint and the comment-history gate.

@kyleseaman
kyleseaman force-pushed the crew/devmate/otel-pr3 branch from 7ff1b31 to e4c160a Compare November 13, 2026 04:12
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Nov 13, 2026
@KiroCrewCaptain

Copy link
Copy Markdown
Collaborator

Linked #1643 is a T3 item still waiting on a maintainer design sign-off, so this cannot merge until a maintainer records it. A maintainer will post the decision here.

Review result on this head: no blocking findings. The review followed the cron path through _execute_with_timeout and fair admission, where handed_over() adopts the root before its first await. It checked that TurnDriver.run now delegates unchanged to _drive, and that the approval-floor pin moved with it at the same count. It also checked that the new span attributes use closed vocabularies or ids. The PR's own tests pass, and deleting nine of the invariant lines it adds makes a named test fail, including both adopt_root admission tests. Good work reusing the phase 1b run machinery, so a cron run gets the same anchor, outcome and roll-up tree as a sub-agent run.

Optional:

  • cron_service/service/run_execution.py:77: no test covers the claim.cancelled branch of _run_status. Deleting it changes no test result, and a user-cancelled script run that returns normally would then be traced as failed. An engine test that sets claim.cancelled = True and job.last_status = "error" and returns normally would pin it.
  • shared_crew/admission_entry.py:516: no test covers the new dispatch.close() on a refused or ended admission. The body also says main left that dispatch un-awaited, but main passed dispatch straight to the gate, and the gate closed it. The close is needed now because the gate closes the handed_over() wrapper instead. A refusal test and a one-line body fix would cover both.
  • docs/request-for-change/rfc-otel-tracing.md:598: the RFC still shows one unsplit phase 3 and a cron <job> anchor keyed by job id, run id and fire time. Recording the 3a/3b split and the kind-named (job id, marker_run) anchor there would help the phase 3b review.

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve. I read the whole diff on this head, tests included. All checks are green.

Scope. This is RFC §10 row 3, split into 3a (this PR) and 3b (workflows). It is the next phase of the design the RFC's owner accepted, and it lands PR by PR like 1a, 1b and 2. It is about 650 lines of product code and docs, well inside row 3's ~950 budget, and the rest is tests.

Data leaving the host. There is no new export, destination, config key or setting. Everything stays behind the existing telemetry.traces_enabled switch, which defaults to off, and leaves only through the phase 2 OTLP exporter when a collector is named. The new attributes are a closed vocabulary added to ALLOWED_KEYS:

  • kirocrew.trigger (six constants, and an unknown value raises).
  • kirocrew.channel (only a CHANNEL_SESSION_NAMESPACES value, never a renderer label outside it).
  • kirocrew.cron.job_kind (script, command or agent).
  • kirocrew.cron.job_id, raw like the other ids. The anchor is named by job kind, never the job name, and a test checks the name stays out of every span.

Off path. With tracing off, surfaces.root is one context-variable write. test_off_a_root_imports_no_sdk checks that OpenTelemetry is never imported.

Behavior. _open_run and _close_run catch every error and log it at DEBUG, so tracing cannot fail a cron run. TurnDriver.run only splits into run and _drive. The cron body under _execute_with_timeout is the same code, indented under the root. admitted_cron_dispatch now wraps the dispatch so the admitted task adopts the run's root, and it closes a dispatch that never started instead of leaving it un-awaited. Nothing new is refused, and there is no regex.

Phase 3a direction signed by maintainer (bolichen97) as the RFC's row 3. Phase 3b (workflow roots) gets its own review.

@bolichen97
bolichen97 merged commit e1186ca into main Nov 13, 2026
97 of 112 checks passed
@bolichen97
bolichen97 deleted the crew/devmate/otel-pr3 branch November 13, 2026 09:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

readiness: passed Eligible automated validation passed for the current revision

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants