Skip to content

feat(tracing): queue wait and the links a send, a broadcast and a workflow launch leave (#1643) - #37158

Merged
bolichen97 merged 1 commit into
mainfrom
crew/devmate/otel-pr4b
Nov 14, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
crew/devmate/otel-pr4b

Conversation

@kyleseaman

@kyleseaman kyleseaman commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: With tracing on, a turn that waited in a session's queue shows how long it waited and which turn sent it, and a workflow run shows the tool call that launched it.

Today a queued dashboard turn starts a fresh trace with nothing saying how long it sat in the queue. A worker's turn started by its conductor's session_send or session_broadcast has no link back to the conductor's turn, and a workflow run's root points at no tool call, so a reader cannot walk from the run back to the workflow_run call that started it.

Why it matters

The tracing RFC's §5.3 names session_send, broadcast and workflow launch as the asynchronous causes a trace links to, and the queued sender as one of them. Without the links, an Ultra Mode or Bot conductor's dispatches and the worker turns they cause are unrelated traces. Without the wait, a turn that sat behind a long one looks slow when it was only queued.

Not a goal

  • Steer links and the conductor's ledger writes (a later Ultra Mode phase).
  • Persisting queue wait across a restart: entries keep their sender, not their wait.
  • New config keys or settings UI.

What changed (motivation → approach → change)

With tracing on, a queued turn records how long it waited and links to the turn that sent it, and a workflow run links to the tool call that launched it.

Design record: the native tracing RFC §5.3, §10 row 4. 4a (#37005) added the steps inside a turn; this is 4b.

Approach: the sender, or the launcher, is named where the cause is still visible, and the turn or run it causes links back to it when its span is created, so every link is set at span start.

  • Queue wait. dashboard/slot_queue_repository.py::queue_append notes each entry's append time in memory (steps.note_queued): not on the entry, whose dict shape is compared wholesale. chat_turn/queue_drain.py hands the drained entries to the turn's task (steps.drained), and dashboard/trace_binding.py::bind_turn takes them once for a top-level turn: kirocrew.queue.wait_ms, from the oldest entry's append, on the turn span only (a new span_attrs on turns.bind_turn_root).
  • Send and broadcast links. session_control_parts/session_messaging.py::send_to_target (which broadcast also uses) passes the caller's session key on session_deliver.peer(sender=). dashboard/session_deliver.py::_deliver_peer names the sender's open session_send/session_broadcast tool span, else its turn span: on the queued entry as meta.trace_sender, and in the context a turn started at once copies. The receiving turn takes one link per sender.
  • Workflow launch. tracing/workflows.py::pin_launcher keeps the open workflow_run tool span (else the turn), sets kirocrew.launched_workflow_run_id on it, and the run's anchor links to it through a new links= on tracing/surfaces.py::root. The completion turn links to the same span.
  • Tool roles. turns.py's spawn-tool match becomes a role match (spawn, send, workflow) with role_span, the one open tool span of a role.
  • Allowlist: kirocrew.queue.wait_ms, kirocrew.launched_workflow_run_id.
  • Spec: metrics.md Traces, in this commit.

Rejected: a link on the sender's tool span pointing forward (the receiving turn does not exist yet when that span starts), and a per-slot "next turn's sender" record, which would hand a sender to whatever turn ran next.

Backwards compatibility

Compatible: tracing is off by default. Off, each new call returns at the switch read, and the drain and a peer delivery set one context variable each; meta.trace_sender is written only while tracing is on, and the queue's wire view (queue_entry_view) lists only its own keys, so the dashboard never shows it.

Removes nothing: the diff adds a link, two allowlisted attributes, an optional sender on session_deliver.peer and an optional links on surfaces.root.

Tests

test/tracing/test_tracing_steps.py:

  • A queued turn links to its sender's session_send tool span and carries kirocrew.queue.wait_ms == 3000 (the oldest of two drained entries, under a ManualClock), on its turn span only; it is bound by the real bind_turn.
  • The real _deliver_peer stamps the sender on the queued entry beside the caller's meta, and a turn it starts at once links to the sender.
  • The real SlotQueueRepository.queue_append starts an entry's wait, and the hand-over is taken once.
  • A malformed sender reference links nothing.

test_tracing_workflows.py: the run's anchor links to the workflow_run tool call that launched it, in a separate trace, and that span carries kirocrew.launched_workflow_run_id. test_tracing_runs.py: send, broadcast and workflow tool names match under any harness prefix. test_off_imports_no_sdk_and_starts_no_thread also drives the send and queue calls with tracing off. The pinned allowlist gains the two keys.

Manual verification

N/A for manual steps: the tests drive the real delivery, queue and dashboard binding with an in-memory exporter; no UI changes.

Local pre-push evidence, on main 6a61c8465f: test/tracing plus the queue drain, session send, broadcast, session deliver and dashboard composition test files (1,032 tests) pass, and test/tracing passes with the otlp extra installed. mypy on the whole package, flake8, isort, black, semgrep and the flake ratchet pass.

Related Issues

Part of #1643 (phase 4b; follows #37005).

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@kyleseaman
kyleseaman requested review from a team and KiroCrewCaptain as code owners November 14, 2026 08:47
@github-actions github-actions Bot added the readiness: action required A blocking check or review needs attention label Nov 14, 2026
@github-actions

github-actions Bot commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 0fc53673c69ecdf2b0fc88f8a4189bbabf2f4b3b touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 0fc53673c69ecdf2b0fc88f8a4189bbabf2f4b3b — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks. Each cause is named where it is still visible and linked when the span starts. With tracing off the change is inert, and the renamed private spawn-tool helpers have no remaining readers.

Notes

  • The wait start is kept only in memory, in a table capped at 4096 entries that drops the oldest first. If more than that many entries are waiting at once, the oldest lose their kirocrew.queue.wait_ms without any sign. The fault is reversible, but the cap could be named in the metrics.md "Queue wait" paragraph.
  • role_span falls back to the turn span when more than one session_send call is open. A conductor that runs sends in parallel then gets links to the turn, not to the call. Say so in the spec, as spawn_span_of already does.

[DESIGN-REVIEWED] 0fc5367

Description read

[DESCRIPTION-READ] d956bf2c2563eb0ddebe163670ffee2f8ee3dc75c3975ea0e76173b13ac4e029

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of 0fc53673c69ecdf2b0fc88f8a4189bbabf2f4b3b — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Nothing blocks. Every item comes from the base partial RFC (rfc-otel-tracing §5.3, §10 row 4: "queue wait, … send, broadcast and workflow-launch links"), and tracing is off by default.

What this change ships

Inventory (8 items) — 8 justified

Intent: with tracing on, a reader can see how long a queued turn waited, and can walk from a worker's turn or a workflow run back to the call that caused it. This is an ADDITION.

  1. The turn span of a queued top-level turn carries kirocrew.queue.wait_ms, measured from its oldest entry — justified
  2. A turn started by session_send or session_broadcast links to the sender's tool span, or to the sender's turn when no tool span is open — justified
  3. A queued peer entry carries meta.trace_sender while tracing is on — justified
  4. A workflow run's anchor links to the workflow_run tool call that launched it — justified
  5. The completion turn links to that tool span instead of the launching turn — justified
  6. The launching workflow_run span carries kirocrew.launched_workflow_run_id — justified
  7. peer(sender=) and surfaces.root(links=) are new optional parameters, one caller each — justified
  8. The spawn-tool match becomes a role match (spawn, send, workflow) with role_span; spawn_span_of now uses it — justified

[FIRST-PRINCIPLES-REVIEWED] 0fc5367

Description read

[DESCRIPTION-READ] eabb22f225a5e79b937af369060b13acc5e6490b14cdffdeb2f463ebca016223

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 0fc53673c69ecdf2b0fc88f8a4189bbabf2f4b3b — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 0fc5367

Verdict parsed from the review's SHA-scoped output markers for commit 0fc53673c69ecdf2b0fc88f8a4189bbabf2f4b3b.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 0fc53673c69ecdf2b0fc88f8a4189bbabf2f4b3b false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 0fc53673c69ecdf2b0fc88f8a4189bbabf2f4b3b and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 0fc5367

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 0fc53673c69ecdf2b0fc88f8a4189bbabf2f4b3b false-positive: <reason, 20 to 500 characters>

@kyleseaman
kyleseaman force-pushed the crew/devmate/otel-pr4b branch from f499c62 to 0fc5367 Compare November 14, 2026 11:06
@kyleseaman

Copy link
Copy Markdown
Contributor Author

Rebased: main fixed the two red tests this head carried (f499c62 -> 0fc5367, base 12a03ac -> 6a61c84); CI re-running. test/tracing/test_tracing_otlp.py::test_a_traced_turn_reaches_a_loopback_receiver_with_its_vault_header and ::test_the_exit_flush_sends_spans_still_queued failed on base 12a03ac. #37171 fixes both on main, and a rerun reuses the old merge commit. test/tracing passes locally with the otlp extra installed. This PR's own diff is unchanged.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Nov 14, 2026
@KiroCrewCaptain

Copy link
Copy Markdown
Collaborator

Code change approver recommendation: A person should decide

Why: the decider raised 2 possible problems and estimates a 55% chance that this change introduces a defect; a person should check them.

The tracing links, the queue-wait attribute and the role-span refactor are mostly sound, and the hand-over is taken once. But the new queue-meta key trace_sender is never removed by the queue drain, which copies entry meta onto the persisted transcript user row. With tracing on, the trace/span id lands in saved transcripts and in the client payload, contrary to the PR's claim that the dashboard never shows it. No test covers it. Requesting changes: remove META_SENDER in the drain and add a test of the persisted row.

Things to check (possible problems the decider raised; at this confidence many turn out to be fine):

  • With tracing on, every queued session_send/broadcast writes trace_sender: <trace id>:<span id> into the receiving session's persisted transcript user row, and row meta goes out to the client through _redact_meta_for_role. The PR says meta.trace_sender is only queue plumbing that the dashboard ne (src/kiro_crew/dashboard/chat_turn/queue_drain.py, medium)
  • No test pins that the new trace meta key stays off the transcript row, and that test would have caught the leak above. (test/tracing/test_tracing_steps.py, low)

An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit 0fc53673c6 · 474 changed lines. If this recommendation is wrong, reply here: replies are how the approver is evaluated.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Nov 14, 2026
@KiroCrewCaptain

Copy link
Copy Markdown
Collaborator

This changes behaviour users already rely on, so it needs a design sign-off from a maintainer before it can merge. A maintainer will post the decision here. To propose a design, follow the request-for-change process in CONTRIBUTING.md (docs/request-for-change/).

Review result on this head: no blocking finding. We read the full source diff and traced each link to where it is set. Both send_to_target call sites pass the caller's session key. The run arm's task copies the context that sending sets, and the queue arm carries the sender in meta. bind_turn takes the queue wait only once. pin_launcher reads the open workflow_run span before the run is scheduled. The new tests drive the real _deliver_peer, queue_append and bind_turn under an injected clock, with no sleeps. With tracing off, every new call returns at the switch read. Well done: each link is captured where its cause is still visible and set when the span starts, which avoids the misattribution a per-slot "next sender" record would cause.

Optional:

  • queue_drain.py around line 1238 copies the entry meta onto the drained row, and trace_sender is not one of the popped plumbing keys. So the sender's trace and span id is saved on the receiver's transcript row and sent to the client. That contradicts the body's "the dashboard never shows it". One pop next to SESSION_SEND_QUEUED_META, plus a test that drains a queued peer send and checks the row meta, would fix it. steps.drained reads the consumed entries, so the link still works.
  • Entries added through queue_insert are never passed to note_queued, so their kirocrew.queue.wait_ms is missing or too low. Time them too, or say so in the metrics.md "Queue wait" paragraph.
  • No added test runs the real drain or send_to_target, so deleting trace_steps.drained(consumed) or sender=caller_session_key breaks no test. The drain test above could also assert the sender link and the wait.

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve. Read the whole diff (queue_drain.py, session_messaging.py, session_deliver.py, slot_queue_repository.py, trace_binding.py, tracing/correlation.py, steps.py, surfaces.py, turns.py, workflows.py, the metrics spec and the tracing tests).

  • Tracing only. With tracing off, note_queued, sending and take_queued return at the provider.peek() read, and drained / sending still reset their hand-over, so no turn inherits a stale one (test in test_tracing_turns.py).
  • Delivery behaviour is unchanged: peer(steer=..., sender=...) defaults sender to "", and _deliver_peer only adds one trace_sender meta key next to the caller's own queue meta.
  • The registries are bounded (_MAX = 4096 queued times, _MAX_LINKS = 64 links). A malformed sender reference makes no link; it is checked as two hex ids with plain string checks, not a regex (test).
  • _is_spawn_tool becomes _tool_role with the same name match; spawn parenting reads the same through role_span.
  • Two allowlist keys are added (kirocrew.queue.wait_ms, kirocrew.launched_workflow_run_id), both ids or numbers, never text; the pin test is updated.
  • No new refusal, no API response change.

Checks: 97 ok, 0 red, 0 pending on this head.

@bolichen97
bolichen97 merged commit 40a4060 into main Nov 14, 2026
101 of 109 checks passed
@bolichen97
bolichen97 deleted the crew/devmate/otel-pr4b branch November 14, 2026 23:24
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Nov 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants