Skip to content

perf(chat): per-turn timing markers behind KIROCREW_TURN_TIMING - #29417

Merged
bolichen97 merged 1 commit into
mainfrom
pearcekieser/perf-turn-timing-markers
Nov 5, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
pearcekieser/perf-turn-timing-markers

Conversation

@Pearcekieser

@Pearcekieser Pearcekieser commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A load-bed run can show where a chat turn spends its time before the first model byte, so the next scale fix targets the wait that actually dominates.

Under the 10x scale bed the first-byte p50 is 66 s. That is consistent with a herd of 300 chats cold-starting through the start gates (17 outer cold-start permits, 8 inner spawn-admission permits), but nobody could check it. The gate-wait lines (acp_cold_start stage=queue_wait, session_cold_start stage=queue_wait) are logged at INFO, the gateway defaults agent.log_level to WARNING, and the bed runs at that default, so no queue-wait line was ever read. Several waits also sit before the session claim and are not timed at all: the boot memory-preparation wait (up to 30 s), the session dispatch lock, and about sixteen asyncio.to_thread hops on the default executor.

Why it matters

Three different fixes are on the table: more start permits, a cheaper spawn and session/new, or a dedicated turn executor. Each one costs a PR and a bed run. Without a breakdown of the first-turn p50, the choice is a guess, and a wrong guess spends a fix cycle on a wait that is not the bottleneck.

Not a goal

  • Changing any gate size, executor or admission order. This PR only measures.
  • The dedicated mc-turn executor and the registry-lock crumb write in session allocation. Both wait on the numbers this PR produces.
  • Parsing the turn_timing lines in the bed's run_bed.py. That belongs to the bed PR.

What changed (motivation → approach → change)

Each turn gets one timing record (new src/kiro_crew/turn_timing.py), but only when KIROCREW_TURN_TIMING=1 is set. The record opens in the /api/chat handler (chat_api/chat_send.py, with the import chat_handlers.py needs because chat_send runs on its globals) and rides a context variable into the turn task. The runner (chat_runner.py) adopts it at entry and binds it to the turn's own task, so a task the turn starts, such as a warm-pool refill, does not record into it, and chat_turn/turn_admission.py marks the spans, before memory preparation, the dispatch lock and the claim, because those waits come before the claim. The three start gates already report every acquire through notify_start_queue in start_priority.py, and the provider start in providers/acp_runtime_start.py already measures its phases. The record reads both, so nothing is timed twice. At the first visible model output, or when the turn ends without one, the gateway writes one turn_timing outcome=... line to gateway.log. That line carries the per-span times, the three gate waits, the provider phases, the default executor's queue depth at turn start and at claim, and is_new, resumed and priority. The flag raises only the kiro_crew.turn_timing logger to INFO, so turning it on does not enable every other INFO line and change the load being measured.

Nothing records outside the flag. Earlier rounds of this PR kept always-on per-gate wait counts, first as counters on /api/status and then as a histogram on the metrics recorder. Both are gone: the per-turn line already carries all three gate waits, and the first get_recorder() call of a process builds the recorder on the calling thread, which on the spawn-admission path is the event loop (test_acp_spawn_offload.py caught it).

Nothing new goes on the wire. A bed run that wants the acknowledgement time apart from the first model byte can read it from the response headers: resp.prepare sends them right after the turn is dispatched, before any chunk, so the driver stamps turn_ack_secs when session.post returns.

Backwards compatibility

Compatible: no frame, row, route, metric or response shape changes, and the timing record does nothing unless the variable is set.

Removes nothing: no field, route, row or log line is removed or renamed.

Tests

test/test_turn_timing.py:

  • With the flag unset, importing the module leaves the logger level at NOTSET and no marker line is logged. With KIROCREW_TURN_TIMING=1 the level is INFO.
  • A runner turn on the scripted ACP double logs exactly one outcome=visible line carrying every named field. memory_wait_ms reads exactly 250.0 under a fake clock that advances only inside the memory-preparation wait. No broadcast frame or slot row carries a wait or a depth.
  • Gate waits reported through notify_start_queue, and provider phases, land on the turn's line. A turn that pays two provider starts reports the sum of each, so spawn_init_ms - spawn_admission_wait_ms stays non-negative. A task the turn starts records nothing into the turn's line.
  • A nested /prompts get turn, which runs in the outer turn's task and context, writes none of its admission, gate waits or phases into the outer turn's line. The record is hidden from it with turn_timing.suspend() and given back when it ends.
  • A gate acquire never reaches the metrics recorder, with the flag on or off (get_recorder patched to fail).
  • POST /api/chat returns 200 and streams its one assistant row with the flag on and off, writes one sse_first_frame line only with the flag on, and the SSE body carries no wait or depth. Without the turn_timing import on chat_handlers this test fails, because chat_send runs on that module's globals.

test/scale_budget/test_turn_admission_work.py::test_turn_path_admission_is_budgeted replaces its scale-F16: strict xfail placeholder with the real budget, and the marker's row leaves MARKERS.md. A send through the real _run_chat (the scripted session double from chat_runner_coverage_helpers) costs 0 session listings and 0 process-tree walks. The first send costs at most 1 agent-catalog refresh, the cold snapshot's, and a second send on the same slot costs none. With dispatch_kiro_agent forced to refresh on every send, the second-send assertion fails.

Determinism: test/test_turn_timing.py and the new budget test passed 20 times at -n0, 10 times at -n 4 beside test_chat_send_echo_scope.py, test_memory_admission_queue.py and test_handlers_system_meminfo_unavailable.py and the rest of test/scale_budget, and in 3 shuffled orders.

Run: pytest test -k "start_priority or chat_send or turn_admission or chat_runner or turn_timing" plus the status and streaming tests passed (1923 tests). The kirocrew-client-py suite passed (85 tests), and test/scale_budget passed (14 passed, 13 expected failures).

Both local review briefs, gpt and opus, ran on Claude Opus 5.5 because the local CLI serves no GPT model, so the gpt brief ran at reduced fidelity. Both flagged a 50 ms timer in the memory-wait test, which a fake clock now replaces.

Manual verification

How a bed run reads this. Start the gateway with KIROCREW_TURN_TIMING=1. Each turn leaves one turn_timing outcome= line with key=value fields; split first turns (is_new=True) from later turns and take p50 and p90 per field. Pure spawn time is spawn_init_ms - spawn_admission_wait_ms, and pure session/new time is session_new_ms - session_new_wait_ms. Count starts from these lines or from acp_cold_start stage=spawn lines, never from session_cold_start stage=queue_wait, which logs only waits of 100 ms or more. Parsing into per-field p50 and p90 in run_bed.py is the bed PR's follow-up.

The decision rule for the first behaviour PR: whichever of these exceeds half of the first-turn p50 picks the fix. If the three gate waits do, the fix is more start permits. If pure spawn plus session/new does, the fix is a cheaper start. If the memory wait plus the executor-bound spans do, the fix is the mc-turn executor.

Related Issues

Fixes #29416

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@Pearcekieser
Pearcekieser requested review from a team and KiroCrewCaptain as code owners November 4, 2026 00:57
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 4, 2026
@github-actions

github-actions Bot commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 937ccccfef9b43ad78a80c40c1b064c7084ba140 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of 937ccccfef9b43ad78a80c40c1b064c7084ba140 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Nothing blocks: the timing record stays off unless KIROCREW_TURN_TIMING is set, and it reads the gate waits and provider phases the code already measures instead of timing them a second time. One logged line is something the bed's decision rule never reads.

Notes

  • Item 3 — oversized: the description's decision rule ("whichever of these exceeds half of the first-turn p50 picks the fix") never uses stage=sse_first_frame, and the description already says the ack time comes from response headers ("the driver stamps turn_ack_secs when session.post returns"). Subtraction: delete turn_timing.sse_first_frame, its call in chat_send.py, the sse_logged key, and the sse_lines assert. Clears when: the mechanism is removed, or a human amends the Goal.

What this change ships

Inventory (6 items) — 4 justified

Intent: let a load-bed operator see where a chat turn's pre-first-byte time goes, so the next scale fix targets the wait that dominates. ADDITION (measurement only).

  1. With KIROCREW_TURN_TIMING=1, each top-level turn writes one turn_timing outcome=… line to gateway.log with span, gate-wait, phase and executor-depth fields. — justified
  2. Only the kiro_crew.turn_timing logger is raised to INFO, so other INFO lines stay off and the measured load does not change. — justified
  3. SSE requests also log a stage=sse_first_frame line. — oversized
  4. Every start-gate acquire in notify_start_queue now reports its wait to the open turn record, including when no caller callback is wired. — justified
  5. Provider start phases (spawn_init, session_load, session_new, set_model) are added to the turn's record. — justified
  6. The scale-F16 strict-xfail placeholder becomes a real turn-path admission budget test, and its MARKERS.md row is removed. — rides along

[FIRST-PRINCIPLES-REVIEWED] 937cccc

Description read

[DESCRIPTION-READ] 91c64a05ff897f18ad3426030a9c98250d8b883171e96b3265f60b9272de8cd9

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 937ccccfef9b43ad78a80c40c1b064c7084ba140 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: the measurement is off by default, logs only, and a revert removes it cleanly. It reuses the timings the gates and provider phases already report, so nothing is timed twice.

Notes

  • notify_start_queue decides a call is an acquire by len(args) == 2, so it relies on the order of the callback's arguments. If a gate passes (queue, wait_ms) the other way round, or adds a third argument, its wait silently drops off the turn_timing line instead of failing. Have each gate pass the wait and the queue name as named arguments at its acquire, so a mismatch fails loudly.

[DESIGN-REVIEWED] 937cccc

Description read

[DESCRIPTION-READ] c10a21b2f880d78328bff25e50432e4631d60936796dbe0c867af859090a9a42

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 937ccccfef9b43ad78a80c40c1b064c7084ba140 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 937cccc

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 937ccccfef9b43ad78a80c40c1b064c7084ba140 false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 937ccccfef9b43ad78a80c40c1b064c7084ba140 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 937cccc

Verdict parsed from the review's SHA-scoped output markers for commit 937ccccfef9b43ad78a80c40c1b064c7084ba140.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 937ccccfef9b43ad78a80c40c1b064c7084ba140 false-positive: <reason, 20 to 500 characters>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 4, 2026
@Pearcekieser Pearcekieser changed the title perf(chat): turn timing markers, accepted SSE row and queue-wait counters perf(chat): turn timing markers and queue-wait counters Nov 4, 2026
@Pearcekieser
Pearcekieser force-pushed the pearcekieser/perf-turn-timing-markers branch from 497509a to b9c096b Compare November 4, 2026 03:16
@Pearcekieser

Pearcekieser commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor Author

span=81ea91c8d9ad
self-added: yes
mechanism: always-on accepted SSE row on /api/chat
resolution: removed

fixed in b9c096b: the always-on {"cls": "accepted"} row is removed, along with its write, the chat_handlers re-export, the client skip, the client stub row and the api-reference sentence. /api/chat frames now match main.
A bed run that needs ack time apart from the first model byte reads it from the response headers. aiohttp StreamResponse.prepare() sends them right after dispatch, so the load driver can stamp turn_ack_secs when session.post returns. The PR body says so.

@Pearcekieser Pearcekieser changed the title perf(chat): turn timing markers and queue-wait counters perf(chat): turn timing markers and a start-queue wait histogram Nov 4, 2026
@Pearcekieser
Pearcekieser force-pushed the pearcekieser/perf-turn-timing-markers branch from b9c096b to a0ba3d1 Compare November 4, 2026 04:57
@Pearcekieser

Pearcekieser commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor Author

span=690b3aa5f009
self-added: yes
mechanism: hand-built per-gate wait counters on /api/status
resolution: removed

fixed in a0ba3d1: the counters, their lock and buckets, and the turn_queue_waits key on /api/status are gone; handlers_system.py matches main.
Each gate acquire now records one kirocrew.session.start_queue.wait (ms) observation on the existing recorder, with a closed gate attribute. The name is in metrics/events.py, the bounds are _STARTUP_BUCKETS_MS in metrics/provider.py, and the metrics spec lists it.

@Pearcekieser

Pearcekieser commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor Author

span=87896f260ac2
self-added: yes
mechanism: per-turn provider phase field kept last-wins

fixed in a0ba3d1: turn_timing.phase() now adds a repeated phase to the earlier value, the same way gate_acquired() sums waits, so a turn that claims a replacement session reports both starts in both fields and spawn_init_ms - spawn_admission_wait_ms stays non-negative.
Regression test: test_a_second_provider_start_sums_its_phases_like_its_waits (6000 + 500 ms phases, 5000 + 100 ms waits, expects 6500.0 and 5100.0).
Repair ran through a model-pinned kiro-cli delegate requesting claude-opus-5.5, because no GPT model is served there; served model unverified.

@Pearcekieser

Pearcekieser commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor Author

span=8a080d8950ca
self-added: no
mechanism: provider phase boundaries in _start_kiro_runtime_impl

Rebutted: true but out of goal. The replacement spawn after a failed session/load runs after _t_sess starts, so it lands in phases["session_new"]. That boundary is the provider's, and main's kirocrew.session.startup.duration phase=session_new already reports it the same way.
This PR reuses the provider's phase values on purpose rather than timing a phase twice. Moving the boundary would change the existing startup metric's phase split, and the path only runs when a runtime dies during session/load, which is rare next to the cold-start herd the bed measures. Adjudication also downgraded it.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author

Electron Shell Tests is red on main too: the same two linux-desktop-integration.real-fs.test.js tests fail on main run 41766971044 (b89a45d). It is tracked in #29439, and #29444 and #29445 fix it. This PR waits for that fix and rebases onto it.

@Pearcekieser
Pearcekieser force-pushed the pearcekieser/perf-turn-timing-markers branch 2 times, most recently from 077185c to fa7eccc Compare November 4, 2026 09:16
@Pearcekieser

Pearcekieser commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor Author

span=2765c59cd85c
self-added: yes
mechanism: per-turn timing record carried by a ContextVar into child tasks

fixed in fa7eccc: begin/ensure bind the record to asyncio.current_task(), and gate_acquired/phase skip it from any other task, so the warm-pool refill (and any task the turn starts) no longer records into the turn line. Every acquire still feeds the histogram.
Regression test: test_a_task_the_turn_starts_does_not_record_into_the_turn (a child task's 5000 ms wait and 6000 ms spawn_init leave the line at the owner's 100.0 and -); it fails with the check removed.
Repair ran through a model-pinned kiro-cli delegate requesting claude-opus-5.5; served model unverified.

@github-actions github-actions Bot added merge conflict Branch has merge conflicts with its base — author must resolve before merge readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention labels Nov 4, 2026
@Pearcekieser
Pearcekieser force-pushed the pearcekieser/perf-turn-timing-markers branch from fa7eccc to dbc0ad9 Compare November 4, 2026 10:42
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: passed Eligible automated validation passed for the current revision merge conflict Branch has merge conflicts with its base — author must resolve before merge labels Nov 4, 2026
@github-actions github-actions Bot added the readiness: action required A blocking check or review needs attention label Nov 5, 2026
@Pearcekieser
Pearcekieser force-pushed the pearcekieser/perf-turn-timing-markers branch from 09cf479 to 2ec3f56 Compare November 5, 2026 00:57
@Pearcekieser

Copy link
Copy Markdown
Contributor Author

note (non-blocking): thanks for the review. I pushed 2ec3f56 after your approval, because the spawn-offload red on 09cf479 came from this PR and was not a flake.

gate_acquired called emit_histogram on every start-gate acquire, and the first get_recorder() of a process builds the recorder on the calling thread. On the spawn-admission path that thread is the event loop. TestRuntimeSpawnOffLoop run alone fails on the branch and passes on main; it most likely passed your rerun because another test in that worker had built the recorder first.

The fix removes the always-on kirocrew.session.start_queue.wait histogram. gate_acquired now returns unless KIROCREW_TURN_TIMING=1, and metrics/events.py, metrics/provider.py and the metrics spec match main again. The per-turn line still carries all three gate waits, and nothing else in the diff changed.

On the slot key in the log: agreed. It is the dashboard slot id, written only when the bench flag is on.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Nov 5, 2026
@Pearcekieser

Copy link
Copy Markdown
Contributor Author

Backend Tests (Windows) (7) on 2ec3f56 fails the three test_slack_code_channels_events.py tests that are red on main since #29302, tracked in #30095. They fail the same way on Linux on this branch and do not touch this diff. This PR waits for that fix and rebases onto it; the decisions e2e fix (#29458) has merged.

@bolichen97
bolichen97 disabled auto-merge November 5, 2026 01:36
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 5, 2026
@bolichen97

Copy link
Copy Markdown
Contributor

Update on 2ec3f56: the failing tests are three in test_slack_code_channels_events.py (Backend Tests (3.12, 7) and Backend Tests (Windows) (7)). This PR does not touch them. Main has a fix for it now (commit 03dc8a1).

Please update the branch onto main (git rebase origin/main or the Update branch button with rebase) so CI runs again on a head that has the fix. The change itself looks good to me now that the histogram is gone, and I will approve once the checks are green.

@Pearcekieser
Pearcekieser force-pushed the pearcekieser/perf-turn-timing-markers branch from 2ec3f56 to 62db8a8 Compare November 5, 2026 04:56
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Nov 5, 2026
@Pearcekieser

Copy link
Copy Markdown
Contributor Author

note (non-blocking): thanks. 62db8a8 is rebased onto main with the Slack code-channel fix, and two more changes rode along.

test/scale_budget landed on main, and the scale brief says this PR deletes its own scale-F16: strict xfail when it does. test_turn_path_admission_is_budgeted now runs real sends through _run_chat on the scripted session double. The first send costs 0 session listings, 0 process-tree walks and 1 agent-catalog refresh (the cold snapshot), and the second costs nothing. Its row is gone from MARKERS.md.

A local review also found that a /prompts get turn, which runs a nested _run_chat in the same task and context, wrote its admission marks and gate waits into the outer turn's open record. _run_chat now hides the record from a turn that is not timed (turn_timing.suspend()) and gives it back in the same finally. test_a_nested_prompts_get_turn_records_nothing_into_the_outer_line fails without it.

@github-actions github-actions Bot added merge conflict Branch has merge conflicts with its base — author must resolve before merge readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Nov 5, 2026
Behind KIROCREW_TURN_TIMING=1, log one turn_timing line per turn that splits
the time before the first visible output into spans, the three start-gate
waits and the provider start phases, bound to the turn's own task.
@Pearcekieser
Pearcekieser force-pushed the pearcekieser/perf-turn-timing-markers branch from 62db8a8 to 937cccc Compare November 5, 2026 06:01
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: passed Eligible automated validation passed for the current revision merge conflict Branch has merge conflicts with its base — author must resolve before merge readiness: checking Automated validation is still running labels Nov 5, 2026

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving 937cccc. This is a new head after my earlier approval, so I reviewed it again.

What I checked on this head:

  • All 97 checks are green, the readiness label is passed, and Design, First Principles, Opus 5.5 and GPT 6.1 all report pass or no blocking findings. The branch is on current main, so the Slack code channel tests that were red before pass.
  • The metrics histogram that GPT flagged on the first version is not in this head. The start-queue hook now only records into the turn timing record, and gate_acquired returns at once when KIROCREW_TURN_TIMING is off. The new test refuses get_recorder, so a regression back to initializing telemetry on the event loop would fail it.
  • The timing stays server side: the module doc and the tests keep gate depth and wait out of any WebSocket or SSE frame.
  • New in this head: the turn-admission work budget test for the real _run_chat path, which replaces the old strict xfail entry in MARKERS.md. It uses the scripted session double, so it needs no network.
  • Ten files, none for auth, permissions or the sandbox, and no regex is added.

@bolichen97
bolichen97 merged commit 5b29857 into main Nov 5, 2026
98 checks passed
@bolichen97
bolichen97 deleted the pearcekieser/perf-turn-timing-markers branch November 5, 2026 07:09
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Nov 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Time where a chat turn waits before its first model byte

2 participants