Repository navigation
feat(acp): log abnormal turn timing to gateway.log - #35604
Conversation
UX Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing blocks: the change only adds a pass-through wrapper on the two existing turn stream functions, writes rate-limited WARNING lines to the existing log, and is easy to revert. Notes
[DESIGN-REVIEWED] cbb0395 Description read[DESCRIPTION-READ] 40ab290b8582fef1697162554545743cf77e3a932af4f5b8469be2e17764db8e That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS This adds an always-on log line for abnormal turns. Today the per-turn timing record ( Notes
What this change shipsInventory (6 items) — 5 justifiedIntent: when a chat sits on "Thinking" for minutes, leave a record on disk of which part of the turn was slow, so a bug report carries evidence. This is an ADDITION.
[FIRST-PRINCIPLES-REVIEWED] cbb0395 Description read[DESCRIPTION-READ] be184d09032cb679e9a20531b880c6dd292a15d924a553648c6e0265aab8313b That sha256 names the PR description this verdict read. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
Code change approver recommendation: A person should decideWhy: the decider estimates a 12% chance that this change introduces a defect, within the 12% limit for approving on its own, but it changes 566 lines, more than the 400 it may approve on its own. The PR adds a pass-through async-generator wrapper, What I checked against the code:
An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit |
0db2184 to
1df22c7
Compare
|
|
|
All review lanes pass on 1df22c7. Remaining reds are inherited from main and outside this diff: |
1df22c7 to
2a358fd
Compare
2a358fd to
d1d3768
Compare
|
resolution: removed
|
d1d3768 to
762687f
Compare
KiroCrewCaptain
left a comment
There was a problem hiding this comment.
Thanks for this. The record is content-free and every field is bounded. One problem in the new code needs a fix before this can merge.
src/kiro_crew/tracing/turn_anomalies.py:221: the turn end is measured when the stream closes, not atcomplete.- Where it happens.
watched()callstimer.finish()only in itsfinally.finish()then runs_note_gap(_now()). The dashboard runner breaks out of itsasync forloop oncomplete, because_on_completealways returnsBREAK. It never closes the stream. So the generator only closes after the runner has awaited settle, persist, context usage and the Stop hook. - What goes wrong. After
completethere is no open tool and no user ask, so all of that post-turn time counts as model silence. Take a turn where the model answers in 3 s and a Stop hook then runs for 90 s. Hook timeouts allow up to 300 s. That turn logsreasons: ["slow_model_gap"],max_model_gap_s: 90.0andtotal_s: 93.0, which blames the model for a slow hook. Every line written from this path also overstatestotal_s, and a no-output turn can get a falseno_outputthe same way. - Smallest fix. Do what
TurnTrace.wrapintracing/turns.pyalready does: callfinish(None)whencompletearrives, before yielding it. Makefinishrun only once, so thefinallycall does nothing afterwards. Keep finish-in-finally for streams that end withoutcomplete. Add a test that breaks oncomplete, moves the fake clock pastMODEL_GAP_SLOW_SECS, callsaclose(), and asserts that nothing is logged.
- Where it happens.
Optional:
- A normal close is recorded as
"error": "GeneratorExit".reasons()already treatsGeneratorExitandCancelledErroras non-errors, sorecord()could apply the same rule. - Several triggers pass all tests even when removed: the
GeneratorExit/CancelledErrorexclusion,notices(retry_wait, model fallback, MCP init failure),timeout/stale_recover,no_output, the trailing gap and both caps. One small test for each would protect them. _USER_ASK_KINDSand_ASK_ENDING_KINDScould be pinned to theacp.typesconstants, the same way_TOOL_TERMINALalready is.
762687f to
cbb0395
Compare
|
@KiroCrewCaptain thanks, fixed in head cbb0395 (single commit, rebased onto main).
New tests in
Not taken in this PR: the extra per-trigger tests and pinning |
bolichen97
left a comment
There was a problem hiding this comment.
Approve at cbb0395. I read the whole product diff: tracing/turn_anomalies.py and the two watched wraps in client_turn.py and session_provider.py.
- Captain's point is fixed.
watchednow callstimer.finish(None)whencompletearrives, before yielding it, andfinishruns only once. So post-turn work (settle, persist, a Stop hook) no longer counts as model silence or inflatestotal_s, and thefinallycall does nothing afterwards. A normal close also no longer shows as"error": "GeneratorExit":record()applies the same_closedrule asreasons(). - No prompt or secret in the line. A record holds the session key, backend and model (each length-capped), reasons, timings, event and tool-call counts, notice counts, the stop reason and the exception class name. It holds no prompt text, tool data, file paths or exception message. Lines are rate-limited per session (5 a minute) and go to the normal logger.
- Every failure inside the timer is logged at debug, and the turn goes on.
No regex and no refusal, and nothing is live red (the cancelled runs are superseded by newer passes).
KiroCrewCaptain
left a comment
There was a problem hiding this comment.
This head fixes the earlier change request. watched() now calls finish(None) at complete before it yields the event, and finish runs only once, so post-turn work such as settle, persist and the Stop hook no longer counts as model silence. The three new tests cover this: break on complete, a slow turn that ends at complete, and a close before complete that is not an error. A normal close also no longer records GeneratorExit as an error. The two other optional items from the earlier review were declined, which is fine. On this head the review checked that events pass through unchanged, that inner exceptions re-raise untouched, that the two wrapped paths cannot log the same turn twice, that every event kind checked exists in acp/types.py, that every retained field is bounded, and that the record holds no prompt, tool or exception-message content. Done well: the content-free claim is tested directly, because the test asserts that the prompt text and the exception message never reach the log.
Optional:
- The line now shares the 2 MB x 3 rollover of
gateway.log, and the diagnostics bundle carries it only while it is in the live file,.prevor a kept boot log. It would help to say this in metrics.md, and to note on #35594 that the dedicated file was dropped. - Several triggers still pass every test when removed:
notices(retry wait, model fallback, MCP init failure), thetimeoutandstale_recoverstops,no_output, and the trailing gap. One small test each would protect them. - If the user signs in to an MCP server before the model writes anything, that wait is logged as
slow_first_output. You could subtract the ask time, or add a field that shows a user ask was open.
Problem / Motivation
Goal: When a chat sits on "Thinking" for minutes, Crew leaves a small record on disk that says which part of the turn was slow, so the next bug report can carry evidence.
Users keep reporting turns that show "Thinking" for around five minutes. By the time a report arrives nothing says whether the model never answered, a tool ran long, the engine retried, or the turn timed out.
gateway.logtreats slow and normal turns alike and rolls over quickly, so the evidence is usually gone.Why it matters
Every such report today ends in guesswork. A record of only the abnormal turns, with the slow phase and its length, lets a maintainer tell a model wait from a tool wait from a timeout, and a user can attach it from the diagnostics bundle.
Not a goal
What changed (motivation → approach → change)
Both functions every model turn passes through (
acp/session_provider.py::AcpSessionProvider.streamandacp/client_turn.py::stream_events) already hand their event stream to the tracing wrapper. A second wrapper beside it,tracing/turn_anomalies.py::watched, times the turn from the client side and logs one WARNING line,turn-anomaly <json>, only when the turn was abnormal. It is always on and does not depend on the tracing switch.retry_waitstatus, model fallback or MCP start failure was seentimeout,stale_recoverorerror:…, or the stream raisedThe line goes through the normal logger, so in the gateway it lands in
gateway.log:grep turn-anomaly gateway.logfinds it, that log's own rollover bounds it, and the diagnostics bundle already carries it when logs are included. On top of that, one session logs at most 5 lines a minute, and its next line after that carriessuppressed: <n>.A line carries the session key, backend, model, reasons, timings, counts, stop reason and exception class name; never prompt text, tool data or an exception message. Silence while a tool runs, a permission ask waits or an MCP sign-in is pending is not counted as model time. The turn ends at its
completeevent, so post-turn work the consumer does before closing the stream (settle, persist, Stop hook) is not counted; a stream that ends withoutcompleteends when it closes, and a consumer closing or cancelling it is not an error. The metrics spec's Traces section documents it.Engine notices that never reach Crew's event stream (for example "MCP server still connecting") are not counted; their effect still shows as a slow first output.
Backwards compatibility
Compatible: the wrapper yields every event unchanged and swallows its own failures; it only adds log lines.
Removes nothing: the diff only adds a stream wrapper and the lines it logs.
Tests
test/tracing/test_turn_anomalies.py:refusedtool ends the tool wait, a status frame does not end a pending permission ask, and an MCP server start ends a sign-in wait;completeand closes the stream 90 s later logs nothing for a fast turn, and a slow turn'stotal_sstops atcomplete; a stream closed early is not logged as an error;error:stop and a raised stream are logged by class name, without the message;acp.types.TERMINAL_TOOL_STATUSES;turn-anomalytag;Determinism: the new tests passed 5 repeats and a reversed-order run. Related suites (
test/tracing,test/test_diagnostics.py) pass.Manual verification
N/A — unit coverage sufficient: the wrapper is driven by a fake clock over the real event type, and the lines are read from the real logger.
Related Issues
Closes #35594
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)