Repository navigation
feat(tracing): queue wait and the links a send, a broadcast and a workflow launch leave (#1643) - #37158
Conversation
UX Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing blocks. Each cause is named where it is still visible and linked when the span starts. With tracing off the change is inert, and the renamed private spawn-tool helpers have no remaining readers. Notes
[DESIGN-REVIEWED] 0fc5367 Description read[DESCRIPTION-READ] d956bf2c2563eb0ddebe163670ffee2f8ee3dc75c3975ea0e76173b13ac4e029 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Nothing blocks. Every item comes from the base What this change shipsInventory (8 items) — 8 justifiedIntent: with tracing on, a reader can see how long a queued turn waited, and can walk from a worker's turn or a workflow run back to the call that caused it. This is an ADDITION.
[FIRST-PRINCIPLES-REVIEWED] 0fc5367 Description read[DESCRIPTION-READ] eabb22f225a5e79b937af369060b13acc5e6490b14cdffdeb2f463ebca016223 That sha256 names the PR description this verdict read. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
…kflow launch leave (#1643)
f499c62 to
0fc5367
Compare
|
Rebased: main fixed the two red tests this head carried (f499c62 -> 0fc5367, base 12a03ac -> 6a61c84); CI re-running. |
Code change approver recommendation: A person should decideWhy: the decider raised 2 possible problems and estimates a 55% chance that this change introduces a defect; a person should check them. The tracing links, the queue-wait attribute and the role-span refactor are mostly sound, and the hand-over is taken once. But the new queue-meta key Things to check (possible problems the decider raised; at this confidence many turn out to be fine):
An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit |
|
This changes behaviour users already rely on, so it needs a design sign-off from a maintainer before it can merge. A maintainer will post the decision here. To propose a design, follow the request-for-change process in CONTRIBUTING.md (docs/request-for-change/). Review result on this head: no blocking finding. We read the full source diff and traced each link to where it is set. Both Optional:
|
bolichen97
left a comment
There was a problem hiding this comment.
Approve. Read the whole diff (queue_drain.py, session_messaging.py, session_deliver.py, slot_queue_repository.py, trace_binding.py, tracing/correlation.py, steps.py, surfaces.py, turns.py, workflows.py, the metrics spec and the tracing tests).
- Tracing only. With tracing off,
note_queued,sendingandtake_queuedreturn at theprovider.peek()read, anddrained/sendingstill reset their hand-over, so no turn inherits a stale one (test in test_tracing_turns.py). - Delivery behaviour is unchanged:
peer(steer=..., sender=...)defaultssenderto"", and_deliver_peeronly adds onetrace_sendermeta key next to the caller's own queue meta. - The registries are bounded (
_MAX = 4096queued times,_MAX_LINKS = 64links). A malformed sender reference makes no link; it is checked as two hex ids with plain string checks, not a regex (test). _is_spawn_toolbecomes_tool_rolewith the same name match; spawn parenting reads the same throughrole_span.- Two allowlist keys are added (
kirocrew.queue.wait_ms,kirocrew.launched_workflow_run_id), both ids or numbers, never text; the pin test is updated. - No new refusal, no API response change.
Checks: 97 ok, 0 red, 0 pending on this head.
Problem / Motivation
Goal: With tracing on, a turn that waited in a session's queue shows how long it waited and which turn sent it, and a workflow run shows the tool call that launched it.
Today a queued dashboard turn starts a fresh trace with nothing saying how long it sat in the queue. A worker's turn started by its conductor's
session_sendorsession_broadcasthas no link back to the conductor's turn, and a workflow run's root points at no tool call, so a reader cannot walk from the run back to theworkflow_runcall that started it.Why it matters
The tracing RFC's §5.3 names
session_send, broadcast and workflow launch as the asynchronous causes a trace links to, and the queued sender as one of them. Without the links, an Ultra Mode or Bot conductor's dispatches and the worker turns they cause are unrelated traces. Without the wait, a turn that sat behind a long one looks slow when it was only queued.Not a goal
What changed (motivation → approach → change)
With tracing on, a queued turn records how long it waited and links to the turn that sent it, and a workflow run links to the tool call that launched it.
Design record: the native tracing RFC §5.3, §10 row 4. 4a (#37005) added the steps inside a turn; this is 4b.
Approach: the sender, or the launcher, is named where the cause is still visible, and the turn or run it causes links back to it when its span is created, so every link is set at span start.
dashboard/slot_queue_repository.py::queue_appendnotes each entry's append time in memory (steps.note_queued): not on the entry, whose dict shape is compared wholesale.chat_turn/queue_drain.pyhands the drained entries to the turn's task (steps.drained), anddashboard/trace_binding.py::bind_turntakes them once for a top-level turn:kirocrew.queue.wait_ms, from the oldest entry's append, on the turn span only (a newspan_attrsonturns.bind_turn_root).session_control_parts/session_messaging.py::send_to_target(which broadcast also uses) passes the caller's session key onsession_deliver.peer(sender=).dashboard/session_deliver.py::_deliver_peernames the sender's opensession_send/session_broadcasttool span, else its turn span: on the queued entry asmeta.trace_sender, and in the context a turn started at once copies. The receiving turn takes one link per sender.tracing/workflows.py::pin_launcherkeeps the openworkflow_runtool span (else the turn), setskirocrew.launched_workflow_run_idon it, and the run's anchor links to it through a newlinks=ontracing/surfaces.py::root. The completion turn links to the same span.turns.py's spawn-tool match becomes a role match (spawn, send, workflow) withrole_span, the one open tool span of a role.kirocrew.queue.wait_ms,kirocrew.launched_workflow_run_id.Rejected: a link on the sender's tool span pointing forward (the receiving turn does not exist yet when that span starts), and a per-slot "next turn's sender" record, which would hand a sender to whatever turn ran next.
Backwards compatibility
Compatible: tracing is off by default. Off, each new call returns at the switch read, and the drain and a peer delivery set one context variable each;
meta.trace_senderis written only while tracing is on, and the queue's wire view (queue_entry_view) lists only its own keys, so the dashboard never shows it.Removes nothing: the diff adds a link, two allowlisted attributes, an optional
senderonsession_deliver.peerand an optionallinksonsurfaces.root.Tests
test/tracing/test_tracing_steps.py:session_sendtool span and carrieskirocrew.queue.wait_ms == 3000(the oldest of two drained entries, under aManualClock), on its turn span only; it is bound by the realbind_turn._deliver_peerstamps the sender on the queued entry beside the caller's meta, and a turn it starts at once links to the sender.SlotQueueRepository.queue_appendstarts an entry's wait, and the hand-over is taken once.test_tracing_workflows.py: the run's anchor links to theworkflow_runtool call that launched it, in a separate trace, and that span carrieskirocrew.launched_workflow_run_id.test_tracing_runs.py: send, broadcast and workflow tool names match under any harness prefix.test_off_imports_no_sdk_and_starts_no_threadalso drives the send and queue calls with tracing off. The pinned allowlist gains the two keys.Manual verification
N/A for manual steps: the tests drive the real delivery, queue and dashboard binding with an in-memory exporter; no UI changes.
Local pre-push evidence, on main
6a61c8465f:test/tracingplus the queue drain, session send, broadcast, session deliver and dashboard composition test files (1,032 tests) pass, andtest/tracingpasses with theotlpextra installed. mypy on the whole package, flake8, isort, black, semgrep and the flake ratchet pass.Related Issues
Part of #1643 (phase 4b; follows #37005).
Checklist