Repository navigation
perf(chat): per-turn timing markers behind KIROCREW_TURN_TIMING - #29417
Conversation
UX Review (Opus 5.5) — ⏭️ skippedRevision |
First Principles Review (Opus 5.5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Nothing blocks: the timing record stays off unless Notes
What this change shipsInventory (6 items) — 4 justifiedIntent: let a load-bed operator see where a chat turn's pre-first-byte time goes, so the next scale fix targets the wait that dominates. ADDITION (measurement only).
[FIRST-PRINCIPLES-REVIEWED] 937cccc Description read[DESCRIPTION-READ] 91c64a05ff897f18ad3426030a9c98250d8b883171e96b3265f60b9272de8cd9 That sha256 names the PR description this verdict read. Recompute it with |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing blocks: the measurement is off by default, logs only, and a revert removes it cleanly. It reuses the timings the gates and provider phases already report, so nothing is timed twice. Notes
[DESIGN-REVIEWED] 937cccc Description read[DESCRIPTION-READ] c10a21b2f880d78328bff25e50432e4631d60936796dbe0c867af859090a9a42 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
497509a to
b9c096b
Compare
|
span=81ea91c8d9ad
|
b9c096b to
a0ba3d1
Compare
|
span=690b3aa5f009
|
|
span=87896f260ac2
|
|
span=8a080d8950ca
|
077185c to
fa7eccc
Compare
|
span=2765c59cd85c
|
fa7eccc to
dbc0ad9
Compare
09cf479 to
2ec3f56
Compare
|
note (non-blocking): thanks for the review. I pushed 2ec3f56 after your approval, because the spawn-offload red on 09cf479 came from this PR and was not a flake.
The fix removes the always-on On the slot key in the log: agreed. It is the dashboard slot id, written only when the bench flag is on. |
|
Backend Tests (Windows) (7) on 2ec3f56 fails the three |
|
Update on 2ec3f56: the failing tests are three in Please update the branch onto main ( |
2ec3f56 to
62db8a8
Compare
|
note (non-blocking): thanks. 62db8a8 is rebased onto main with the Slack code-channel fix, and two more changes rode along.
A local review also found that a |
Behind KIROCREW_TURN_TIMING=1, log one turn_timing line per turn that splits the time before the first visible output into spans, the three start-gate waits and the provider start phases, bound to the turn's own task.
62db8a8 to
937cccc
Compare
bolichen97
left a comment
There was a problem hiding this comment.
Approving 937cccc. This is a new head after my earlier approval, so I reviewed it again.
What I checked on this head:
- All 97 checks are green, the readiness label is passed, and Design, First Principles, Opus 5.5 and GPT 6.1 all report pass or no blocking findings. The branch is on current main, so the Slack code channel tests that were red before pass.
- The metrics histogram that GPT flagged on the first version is not in this head. The start-queue hook now only records into the turn timing record, and
gate_acquiredreturns at once whenKIROCREW_TURN_TIMINGis off. The new test refusesget_recorder, so a regression back to initializing telemetry on the event loop would fail it. - The timing stays server side: the module doc and the tests keep gate depth and wait out of any WebSocket or SSE frame.
- New in this head: the turn-admission work budget test for the real
_run_chatpath, which replaces the old strict xfail entry inMARKERS.md. It uses the scripted session double, so it needs no network. - Ten files, none for auth, permissions or the sandbox, and no regex is added.
Problem / Motivation
Goal: A load-bed run can show where a chat turn spends its time before the first model byte, so the next scale fix targets the wait that actually dominates.
Under the 10x scale bed the first-byte p50 is 66 s. That is consistent with a herd of 300 chats cold-starting through the start gates (17 outer cold-start permits, 8 inner spawn-admission permits), but nobody could check it. The gate-wait lines (
acp_cold_start stage=queue_wait,session_cold_start stage=queue_wait) are logged at INFO, the gateway defaultsagent.log_levelto WARNING, and the bed runs at that default, so no queue-wait line was ever read. Several waits also sit before the session claim and are not timed at all: the boot memory-preparation wait (up to 30 s), the session dispatch lock, and about sixteenasyncio.to_threadhops on the default executor.Why it matters
Three different fixes are on the table: more start permits, a cheaper spawn and session/new, or a dedicated turn executor. Each one costs a PR and a bed run. Without a breakdown of the first-turn p50, the choice is a guess, and a wrong guess spends a fix cycle on a wait that is not the bottleneck.
Not a goal
mc-turnexecutor and the registry-lock crumb write in session allocation. Both wait on the numbers this PR produces.turn_timinglines in the bed'srun_bed.py. That belongs to the bed PR.What changed (motivation → approach → change)
Each turn gets one timing record (new
src/kiro_crew/turn_timing.py), but only whenKIROCREW_TURN_TIMING=1is set. The record opens in the/api/chathandler (chat_api/chat_send.py, with the importchat_handlers.pyneeds becausechat_sendruns on its globals) and rides a context variable into the turn task. The runner (chat_runner.py) adopts it at entry and binds it to the turn's own task, so a task the turn starts, such as a warm-pool refill, does not record into it, andchat_turn/turn_admission.pymarks the spans, before memory preparation, the dispatch lock and the claim, because those waits come before the claim. The three start gates already report every acquire throughnotify_start_queueinstart_priority.py, and the provider start inproviders/acp_runtime_start.pyalready measures its phases. The record reads both, so nothing is timed twice. At the first visible model output, or when the turn ends without one, the gateway writes oneturn_timing outcome=...line togateway.log. That line carries the per-span times, the three gate waits, the provider phases, the default executor's queue depth at turn start and at claim, andis_new,resumedandpriority. The flag raises only thekiro_crew.turn_timinglogger to INFO, so turning it on does not enable every other INFO line and change the load being measured.Nothing records outside the flag. Earlier rounds of this PR kept always-on per-gate wait counts, first as counters on
/api/statusand then as a histogram on the metrics recorder. Both are gone: the per-turn line already carries all three gate waits, and the firstget_recorder()call of a process builds the recorder on the calling thread, which on the spawn-admission path is the event loop (test_acp_spawn_offload.pycaught it).Nothing new goes on the wire. A bed run that wants the acknowledgement time apart from the first model byte can read it from the response headers:
resp.preparesends them right after the turn is dispatched, before any chunk, so the driver stampsturn_ack_secswhensession.postreturns.Backwards compatibility
Compatible: no frame, row, route, metric or response shape changes, and the timing record does nothing unless the variable is set.
Removes nothing: no field, route, row or log line is removed or renamed.
Tests
test/test_turn_timing.py:KIROCREW_TURN_TIMING=1the level is INFO.outcome=visibleline carrying every named field.memory_wait_msreads exactly 250.0 under a fake clock that advances only inside the memory-preparation wait. No broadcast frame or slot row carries a wait or a depth.notify_start_queue, and provider phases, land on the turn's line. A turn that pays two provider starts reports the sum of each, sospawn_init_ms - spawn_admission_wait_msstays non-negative. A task the turn starts records nothing into the turn's line./prompts getturn, which runs in the outer turn's task and context, writes none of its admission, gate waits or phases into the outer turn's line. The record is hidden from it withturn_timing.suspend()and given back when it ends.get_recorderpatched to fail).POST /api/chatreturns 200 and streams its one assistant row with the flag on and off, writes onesse_first_frameline only with the flag on, and the SSE body carries no wait or depth. Without theturn_timingimport onchat_handlersthis test fails, becausechat_sendruns on that module's globals.test/scale_budget/test_turn_admission_work.py::test_turn_path_admission_is_budgetedreplaces itsscale-F16:strict xfail placeholder with the real budget, and the marker's row leavesMARKERS.md. A send through the real_run_chat(the scripted session double fromchat_runner_coverage_helpers) costs 0 session listings and 0 process-tree walks. The first send costs at most 1 agent-catalog refresh, the cold snapshot's, and a second send on the same slot costs none. Withdispatch_kiro_agentforced to refresh on every send, the second-send assertion fails.Determinism:
test/test_turn_timing.pyand the new budget test passed 20 times at-n0, 10 times at-n 4besidetest_chat_send_echo_scope.py,test_memory_admission_queue.pyandtest_handlers_system_meminfo_unavailable.pyand the rest oftest/scale_budget, and in 3 shuffled orders.Run:
pytest test -k "start_priority or chat_send or turn_admission or chat_runner or turn_timing"plus the status and streaming tests passed (1923 tests). Thekirocrew-client-pysuite passed (85 tests), andtest/scale_budgetpassed (14 passed, 13 expected failures).Both local review briefs, gpt and opus, ran on Claude Opus 5.5 because the local CLI serves no GPT model, so the gpt brief ran at reduced fidelity. Both flagged a 50 ms timer in the memory-wait test, which a fake clock now replaces.
Manual verification
How a bed run reads this. Start the gateway with
KIROCREW_TURN_TIMING=1. Each turn leaves oneturn_timing outcome=line withkey=valuefields; split first turns (is_new=True) from later turns and take p50 and p90 per field. Pure spawn time isspawn_init_ms - spawn_admission_wait_ms, and pure session/new time issession_new_ms - session_new_wait_ms. Count starts from these lines or fromacp_cold_start stage=spawnlines, never fromsession_cold_start stage=queue_wait, which logs only waits of 100 ms or more. Parsing into per-field p50 and p90 inrun_bed.pyis the bed PR's follow-up.The decision rule for the first behaviour PR: whichever of these exceeds half of the first-turn p50 picks the fix. If the three gate waits do, the fix is more start permits. If pure spawn plus session/new does, the fix is a cheaper start. If the memory wait plus the executor-bound spans do, the fix is the
mc-turnexecutor.Related Issues
Fixes #29416
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)