Skip to content

feat(acp): log abnormal turn timing to gateway.log - #35604

Merged
bolichen97 merged 1 commit into
mainfrom
feat/turn-anomaly-log
Nov 12, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
feat/turn-anomaly-log

Conversation

@iamwhatever

@iamwhatever iamwhatever commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: When a chat sits on "Thinking" for minutes, Crew leaves a small record on disk that says which part of the turn was slow, so the next bug report can carry evidence.

Users keep reporting turns that show "Thinking" for around five minutes. By the time a report arrives nothing says whether the model never answered, a tool ran long, the engine retried, or the turn timed out. gateway.log treats slow and normal turns alike and rolls over quickly, so the evidence is usually gone.

Why it matters

Every such report today ends in guesswork. A record of only the abnormal turns, with the slow phase and its length, lets a maintainer tell a model wait from a tool wait from a timeout, and a user can attach it from the diagnostics bundle.

Not a goal

  • No change to the agent engine or its own logs; everything is measured from Crew's side of the stream.
  • No UI change (for example an elapsed-time label on "Thinking").
  • No new setting: thresholds and caps are module constants.
  • No record of normal turns.

What changed (motivation → approach → change)

Both functions every model turn passes through (acp/session_provider.py::AcpSessionProvider.stream and acp/client_turn.py::stream_events) already hand their event stream to the tracing wrapper. A second wrapper beside it, tracing/turn_anomalies.py::watched, times the turn from the client side and logs one WARNING line, turn-anomaly <json>, only when the turn was abnormal. It is always on and does not depend on the tracing switch.

Logs a line when Threshold
first model output (text, thought or tool call) is late, or never came 30 s
longest silence while no tool runs and no user ask is open 60 s
a retry_wait status, model fallback or MCP start failure was seen any
stop reason timeout, stale_recover or error:…, or the stream raised any

The line goes through the normal logger, so in the gateway it lands in gateway.log: grep turn-anomaly gateway.log finds it, that log's own rollover bounds it, and the diagnostics bundle already carries it when logs are included. On top of that, one session logs at most 5 lines a minute, and its next line after that carries suppressed: <n>.

A line carries the session key, backend, model, reasons, timings, counts, stop reason and exception class name; never prompt text, tool data or an exception message. Silence while a tool runs, a permission ask waits or an MCP sign-in is pending is not counted as model time. The turn ends at its complete event, so post-turn work the consumer does before closing the stream (settle, persist, Stop hook) is not counted; a stream that ends without complete ends when it closes, and a consumer closing or cancelling it is not an error. The metrics spec's Traces section documents it.

Engine notices that never reach Crew's event stream (for example "MCP server still connecting") are not counted; their effect still shows as a slow first output.

Backwards compatibility

Compatible: the wrapper yields every event unchanged and swallows its own failures; it only adds log lines.

Removes nothing: the diff only adds a stream wrapper and the lines it logs.

Tests

test/tracing/test_turn_anomalies.py:

  • a normal turn logs nothing and its events pass through unchanged;
  • a 45 s first output logs one line with the timings and without the prompt text;
  • a long tool run or permission wait is not a model gap, a long silence after a tool result is; a refused tool ends the tool wait, a status frame does not end a pending permission ask, and an MCP server start ends a sign-in wait;
  • an oversized backend tool-call id is not tracked;
  • a consumer that breaks on complete and closes the stream 90 s later logs nothing for a fast turn, and a slow turn's total_s stops at complete; a stream closed early is not logged as an error;
  • an error: stop and a raised stream are logged by class name, without the message;
  • the terminal tool statuses match acp.types.TERMINAL_TOOL_STATUSES;
  • a line is one WARNING starting with the turn-anomaly tag;
  • the per-session rate limit drops extra lines, keeps other sessions' budgets, and reports the dropped count.

Determinism: the new tests passed 5 repeats and a reversed-order run. Related suites (test/tracing, test/test_diagnostics.py) pass.

Manual verification

N/A — unit coverage sufficient: the wrapper is driven by a fake clock over the real event type, and the lines are read from the real logger.

Related Issues

Closes #35594

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@iamwhatever
iamwhatever requested review from a team and KiroCrewCaptain as code owners November 11, 2026 21:29
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 11, 2026
@github-actions

github-actions Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision cbb03951e34e869d217a8f57ff221ef5edd4f277 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of cbb03951e34e869d217a8f57ff221ef5edd4f277 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: the change only adds a pass-through wrapper on the two existing turn stream functions, writes rate-limited WARNING lines to the existing log, and is easy to revert.

Notes

  • Unlike traced, watched wraps every turn whether or not anything is logged, so each turn on both stream paths gets one more async-generator layer. That is fine at this size; keep this wrapper the only always-on one rather than stacking more beside it.

[DESIGN-REVIEWED] cbb0395

Description read

[DESCRIPTION-READ] 40ab290b8582fef1697162554545743cf77e3a932af4f5b8469be2e17764db8e

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of cbb03951e34e869d217a8f57ff221ef5edd4f277 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

This adds an always-on log line for abnormal turns. Today the per-turn timing record (telemetry.traces_enabled) is off by default, so a "Thinking for five minutes" report arrives with nothing on disk. Nothing here blocks.

Notes

  • Item 1 — one consumer, generalized: the new module exposes write, TurnTimer and reset_for_testing, but watched is the only production caller of TurnTimer and write. Subtraction: make write and TurnTimer private (_write, _TurnTimer) so no outside caller can start depending on them.

What this change ships

Inventory (6 items) — 5 justified

Intent: when a chat sits on "Thinking" for minutes, leave a record on disk of which part of the turn was slow, so a bug report carries evidence. This is an ADDITION.

  1. Each abnormal turn writes one turn-anomaly <json> WARNING to gateway.log, whether tracing is on or off — one consumer, generalized: write/TurnTimer are public but only watched calls them
  2. The Kiro path's turns (AcpSessionProvider.stream) pass through the new watcher — justified
  3. The Claude and direct-client turns (AcpClient.stream_events) pass through it too — justified
  4. Time spent on running tools, permission asks and MCP sign-in is left out of the model-silence measure — justified
  5. Each session writes at most 5 lines a minute, and the next line after that carries a suppressed count — justified
  6. The metrics spec gains a section on the abnormal turn log — justified

[FIRST-PRINCIPLES-REVIEWED] cbb0395

Description read

[DESCRIPTION-READ] be184d09032cb679e9a20531b880c6dd292a15d924a553648c6e0265aab8313b

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed cbb03951e34e869d217a8f57ff221ef5edd4f277 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] cbb0395

Verdict parsed from the review's SHA-scoped output markers for commit cbb03951e34e869d217a8f57ff221ef5edd4f277.

False positive or not applicable? A repository writer can comment:
/ai-review override fable cbb03951e34e869d217a8f57ff221ef5edd4f277 false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of cbb03951e34e869d217a8f57ff221ef5edd4f277 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] cbb0395

False positive or not applicable? A repository writer can comment:
/ai-review override gpt cbb03951e34e869d217a8f57ff221ef5edd4f277 false-positive: <reason, 20 to 500 characters>

@KiroCrewCaptain

KiroCrewCaptain commented Nov 11, 2026 •

Copy link
Copy Markdown
Collaborator

Code change approver recommendation: A person should decide

Why: the decider estimates a 12% chance that this change introduces a defect, within the 12% limit for approving on its own, but it changes 566 lines, more than the 400 it may approve on its own.

The PR adds a pass-through async-generator wrapper, watched, around the existing traced wrapper on both turn stream paths: AcpSessionProvider._stream_turn and AcpClient.stream_events. When a turn was abnormal, it writes one rate-limited WARNING line, turn-anomaly <json>, through the normal logger. I found nothing that blocks the merge.

What I checked against the code:

  • Pass-through and cleanup: events are passed on unchanged. Shutdown works the same way as in traced: aclosing closes the inner stream, and a consumer closing the stream (GeneratorExit) or cancelling it (CancelledError) is not counted as an error. The timer finishes once, at complete, before the event is passed on, so the consumer's work after the turn is not counted.
  • No new pause before the prompt: nothing extra is awaited before the inner stream is first pulled, so the prompt is still written as soon as traced's docstring says it is.
  • Event names match: every event kind the module recognises matches a constant in acp/types.py. retry_wait matches types_status.WAIT_REASON_RETRY. The tool-finished statuses equal TERMINAL_TOOL_STATUSES, and a test pins that.
  • **No double loggin

An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit cbb03951e3 · 566 changed lines. If this recommendation is wrong, reply here: replies are how the approver is evaluated.

@iamwhatever
iamwhatever force-pushed the feat/turn-anomaly-log branch from 0db2184 to 1df22c7 Compare November 11, 2026 22:02
@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • Anomaly log ships in a crash-reports-only bundle span=2a6f803f1b6a -- fixed

fixed in 1df22c7: the anomaly log and its rolled copies moved inside if include_logs: with gated=True, so collect_bundle(include_logs=False) leaves them out; the test now asserts a crash-only bundle has no turn-anomalies.log* member or skip.

@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • Resume model-gap timing after MCP authorization completes span=4fe3ecb953ac -- fixed

fixed in 1df22c7: mcp_server_initialized and mcp_server_init_failure now end the user-ask wait (single flag kept), so the model silence after an MCP sign-in is counted; test_an_mcp_server_start_ends_the_sign_in_wait covers it.

@iamwhatever

Copy link
Copy Markdown
Contributor Author

Backend Tests shard 5 red on 0db2184 was test/test_wait_discipline_ratchet.py::test_the_scan_matches_known_sites flagging test/test_update_boot_apply.py:335, a file this PR does not touch; it reproduces on main (from #35088). Fix in flight: #35614. Will rebase once it lands if the red persists.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 11, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor Author

All review lanes pass on 1df22c7. Remaining reds are inherited from main and outside this diff: test_wait_discipline_ratchet.py::test_the_scan_matches_known_sites (flags test/test_update_boot_apply.py:335) and test_updates_handlers_facade.py::test_every_name_served_from_outside_the_family_is_visible_to_type_checkers; the second reproduces locally on main's tree. Fix: #35614 (auto-merge armed). Will rebase once it lands.

@iamwhatever

iamwhatever commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor Author

Update: #35614 fixes the facade red only. The wait-discipline ratchet red had no owner, so #35697 fixes it (closes #35696). This PR rebases once both land.

@iamwhatever
iamwhatever force-pushed the feat/turn-anomaly-log branch from 1df22c7 to 2a358fd Compare November 12, 2026 03:18
@iamwhatever

Copy link
Copy Markdown
Contributor Author

Rebased onto main e1b079c, which carries #35625 (fixes both inherited reds); head is now 2a358fd. #35697 closed as redundant with #35625.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Nov 12, 2026
@iamwhatever
iamwhatever force-pushed the feat/turn-anomaly-log branch from 2a358fd to d1d3768 Compare November 12, 2026 04:16
@iamwhatever

Copy link
Copy Markdown
Contributor Author

resolution: removed

  • The log writer follows symlinks into agent-protected policy files span=4fe3ecb953ac -- fixed

fixed in d1d3768: the separate turn-anomalies.log file and its own rotating handler are removed. Anomaly lines now go through the module logger into gateway.log, so this PR opens no file and rotates nothing itself; the symlink-follow and the cross-process rollover race (same span, line 139) both had no remaining site.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 12, 2026
@iamwhatever iamwhatever changed the title feat(acp): log abnormal turn timing to a size-capped file feat(acp): log abnormal turn timing to gateway.log Nov 12, 2026
@iamwhatever
iamwhatever force-pushed the feat/turn-anomaly-log branch from d1d3768 to 762687f Compare November 12, 2026 05:49
@github-actions github-actions Bot removed readiness: action required A blocking check or review needs attention labels Nov 12, 2026
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention labels Nov 12, 2026

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this. The record is content-free and every field is bounded. One problem in the new code needs a fix before this can merge.

  1. src/kiro_crew/tracing/turn_anomalies.py:221: the turn end is measured when the stream closes, not at complete.
    • Where it happens. watched() calls timer.finish() only in its finally. finish() then runs _note_gap(_now()). The dashboard runner breaks out of its async for loop on complete, because _on_complete always returns BREAK. It never closes the stream. So the generator only closes after the runner has awaited settle, persist, context usage and the Stop hook.
    • What goes wrong. After complete there is no open tool and no user ask, so all of that post-turn time counts as model silence. Take a turn where the model answers in 3 s and a Stop hook then runs for 90 s. Hook timeouts allow up to 300 s. That turn logs reasons: ["slow_model_gap"], max_model_gap_s: 90.0 and total_s: 93.0, which blames the model for a slow hook. Every line written from this path also overstates total_s, and a no-output turn can get a false no_output the same way.
    • Smallest fix. Do what TurnTrace.wrap in tracing/turns.py already does: call finish(None) when complete arrives, before yielding it. Make finish run only once, so the finally call does nothing afterwards. Keep finish-in-finally for streams that end without complete. Add a test that breaks on complete, moves the fake clock past MODEL_GAP_SLOW_SECS, calls aclose(), and asserts that nothing is logged.

Optional:

  • A normal close is recorded as "error": "GeneratorExit". reasons() already treats GeneratorExit and CancelledError as non-errors, so record() could apply the same rule.
  • Several triggers pass all tests even when removed: the GeneratorExit/CancelledError exclusion, notices (retry_wait, model fallback, MCP init failure), timeout/stale_recover, no_output, the trailing gap and both caps. One small test for each would protect them.
  • _USER_ASK_KINDS and _ASK_ENDING_KINDS could be pinned to the acp.types constants, the same way _TOOL_TERMINAL already is.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: passed Eligible automated validation passed for the current revision labels Nov 12, 2026
@iamwhatever
iamwhatever force-pushed the feat/turn-anomaly-log branch from 762687f to cbb0395 Compare November 12, 2026 16:57
@iamwhatever

iamwhatever commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor Author

@KiroCrewCaptain thanks, fixed in head cbb0395 (single commit, rebased onto main).

  1. Turn end at complete: watched() now calls timer.finish(None) when the complete event arrives, before yielding it, as TurnTrace.wrap does. TurnTimer.finish runs only once, so the finally call is a no-op after that. Streams that end without complete (errors, cancels) still finish in finally.
  2. Optional item taken: record() now applies the same GeneratorExit / CancelledError rule as reasons(), so a normal close records "error": "".

New tests in test/tracing/test_turn_anomalies.py (all three fail on the old code):

  • a consumer breaks on complete, the clock moves past MODEL_GAP_SLOW_SECS, then aclose(): nothing is logged;
  • a slow turn broken on complete and closed 90 s later logs total_s up to complete and no slow_model_gap;
  • a stream closed before complete logs error: "".

Not taken in this PR: the extra per-trigger tests and pinning _USER_ASK_KINDS / _ASK_ENDING_KINDS to acp.types.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Nov 12, 2026

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve at cbb0395. I read the whole product diff: tracing/turn_anomalies.py and the two watched wraps in client_turn.py and session_provider.py.

  • Captain's point is fixed. watched now calls timer.finish(None) when complete arrives, before yielding it, and finish runs only once. So post-turn work (settle, persist, a Stop hook) no longer counts as model silence or inflates total_s, and the finally call does nothing afterwards. A normal close also no longer shows as "error": "GeneratorExit": record() applies the same _closed rule as reasons().
  • No prompt or secret in the line. A record holds the session key, backend and model (each length-capped), reasons, timings, event and tool-call counts, notice counts, the stop reason and the exception class name. It holds no prompt text, tool data, file paths or exception message. Lines are rate-limited per session (5 a minute) and go to the normal logger.
  • Every failure inside the timer is logged at debug, and the turn goes on.

No regex and no refusal, and nothing is live red (the cancelled runs are superseded by newer passes).

@bolichen97
bolichen97 enabled auto-merge (squash) November 12, 2026 19:05

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This head fixes the earlier change request. watched() now calls finish(None) at complete before it yields the event, and finish runs only once, so post-turn work such as settle, persist and the Stop hook no longer counts as model silence. The three new tests cover this: break on complete, a slow turn that ends at complete, and a close before complete that is not an error. A normal close also no longer records GeneratorExit as an error. The two other optional items from the earlier review were declined, which is fine. On this head the review checked that events pass through unchanged, that inner exceptions re-raise untouched, that the two wrapped paths cannot log the same turn twice, that every event kind checked exists in acp/types.py, that every retained field is bounded, and that the record holds no prompt, tool or exception-message content. Done well: the content-free claim is tested directly, because the test asserts that the prompt text and the exception message never reach the log.

Optional:

  • The line now shares the 2 MB x 3 rollover of gateway.log, and the diagnostics bundle carries it only while it is in the live file, .prev or a kept boot log. It would help to say this in metrics.md, and to note on #35594 that the dedicated file was dropped.
  • Several triggers still pass every test when removed: notices (retry wait, model fallback, MCP init failure), the timeout and stale_recover stops, no_output, and the trailing gap. One small test each would protect them.
  • If the user signs in to an MCP server before the model writes anything, that wait is logged as slow_first_output. You could subtract the ask time, or add a field that shows a user ask was open.

@bolichen97
bolichen97 merged commit 7e243f9 into main Nov 12, 2026
99 of 106 checks passed
@bolichen97
bolichen97 deleted the feat/turn-anomaly-log branch November 12, 2026 19:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

readiness: passed Eligible automated validation passed for the current revision

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Record abnormal turn timing (slow first output, long model silence, retries) in a size-capped log

3 participants