Skip to content

fix(chat): re-send a throttled turn with one copy of the prompt in history - #21907

Merged
bolichen97 merged 1 commit into
mainfrom
fix/throttle-retry-no-stacked-prompt
Oct 18, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
fix/throttle-retry-no-stacked-prompt

Conversation

@iamwhatever

@iamwhatever iamwhatever commented Oct 18, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: When the model provider throttles a chat turn before it answers, the retry leaves one copy of the message in the model's history, waits about one KAS attempt before re-sending, and records how many retries the turn took.

In a matched latency run (opus-5.5, medium effort, 15 warm turns on KAS), 4 turns were throttled. Each retry re-sent the same message into the same KAS session, and KAS kept every failed copy: one turn ended with four copies of its prompt, and the model's next reply began "There are several queued requests…". The ladder re-sent 2.4 s after a failure that KAS had already retried for about 6 s inside the attempt. Nothing in turn_stats or telemetry said a turn was retried.

Why it matters

Throttled turns cost +7.6 to +30.8 s each in that run, and one got no answer. The extra copies stay in the session for good: the model treats them as more requests, and a copy of a first message carries the whole session context (128k chars). Without a retry count, a latency report cannot split a turn's model time into the first attempt and the retries.

Not a goal

  • Other transient errors (5xx, connection drops): they keep the 2 s ladder and the verbatim re-send; only their retry count is recorded.
  • A throttle after output already streamed: it keeps its one continuation, which never re-sends the prompt.
  • The unattended stream_and_collect path (cron, sub-agents, workflows) and the dedicated AcpClient backends.
  • Counting KAS's own internal re-sends: kiro-cli 2.32.0 does not put them on the wire.
  • Any UI change: the footer does not render retries.

What changed (motivation → approach → change)

After N throttles, then success Before After
Copies of the prompt in KAS history N + 1 1
Waits before re-sends 1, 2, 3 2, 4, 8 s (+25% jitter) 6, 12, 24 s (+25% jitter)
turn_stats.retries on the landed row absent N
kirocrew.watchdog.recovery.outcome stall mechanisms only also throttle / transient_5xx, recovered or exhausted

KAS stores the user row before it asks the model, so a failed attempt's row stays, and a plain re-send adds another. KAS can rewind a session in place: _kiro/checkpoint/revertMultiple {sessionId, messageId} hides that row and everything after it, and every prompt reports its row id as a user_message_id_assigned session_info_update. A live probe on kiro-cli 2.32.0 confirmed the model's next answer leaves the reverted prompt out.

The KAS session handle keeps the row id of its latest prompt. Before each verbatim re-send of a throttled turn (the same-model retry and the model-fallback retry), the chat runner calls the provider's new discard_unanswered_prompt. The handle sends the verb only with _meta.kiro.checkpoints advertised and a row recorded (only KAS's update branch records one), only when no later turn or steer landed after that prompt, and once; any other case, or a failed call, sends nothing and the retry goes out with the copy kept. The rewind runs after the backoff and before the existing Stop re-read, so a Stop during the backoff rewinds nothing and a Stop during the rewind still drops the re-send. AcpError.throttle is tagged from the raw frame like connection_failure, and a throttle waits on throttle_retry_delay, the same doubling curve from 6 s. The slot counts re-sends per mechanism (_turn_retries) for the landed row's turn_stats and the recovery-outcome counter; the specs for providers, the turn-stats footer and metrics say so.

Part Where
Row id capture, the rewind verb and its gate acp/session_handle.py, acp/kas_wire.py, acp/types.py
Provider seam (discard_unanswered_prompt) providers/base.py, providers/acp.py, acp/session_provider.py
Throttle tag, predicate and curve acp/transport_errors.py, llm_helpers.py
Ladder, rewind call, count and telemetry dashboard/chat_runner.py, chat_turn/recovery.py, chat_turn/turn_stats.py, state_parts/slot_ladder.py, state_parts/chat_slot.py

Backwards compatibility

Compatible: every new method, field and tag is additive; a backend without the rewind keeps the failed copy, and non-throttle errors retry on the same schedule.

Removes nothing: the change adds a step before a throttle re-send, a longer throttle wait, and two recorded counts.

Tests

  • test/test_dashboard_chat_transient_throttle.py: a fake provider models KAS history (each attempt stores the prompt; the rewind removes the unanswered one). Two throttles then success: one copy, turn_stats.retries == 2, waits 6–7.5 s and 12–15 s, one recovered telemetry point. The same on a crewmate DM slot. A full ladder of throttles: one copy and an exhausted point. The model-fallback re-send rewinds too. A Stop that lands during the rewind still drops the re-send, on the same-model arm and the fallback arm; with the rewind moved after the Stop re-read, both of those tests fail by assertion. Nearby inputs keep the plain retry: a non-throttle 5xx (no rewind, three copies, 2 s and 4 s waits, retries still recorded), a throttle after partial output (no rewind, continuation), and a Stop during the backoff (no re-send, no rewind).
  • test/test_kas_unanswered_prompt_rewind.py: the handle sends _kiro/checkpoint/revertMultiple with the row id from user_message_id_assigned; it sends nothing off KAS, without checkpoints, after a later turn, after a steer, with no row id, or on a second call; a success: false answer reads False; both providers forward, and a client without the method reads False.
  • test/test_crew_log_edge_throttle.py: also zeroes the throttle curve, since its "throttled" error now takes it.
  • test/test_state_parts_facade.py: the new slot default.
  • test/test_acp_bindings.py: AcpProvider.discard_unanswered_prompt joins the frozen member inventory.

Red before the fix: prove.py --base origin/main reverts every production hunk in a throwaway worktree and re-runs the changed tests: PROVEN, an assertion fails with the bug reintroduced (on main the fake history holds three copies after two throttles, and no retries lands).

Determinism: 20 runs at -n0 of the two new files (16 passed each), 10 runs at -n 2 --dist loadgroup --max-worker-restart=2 with test/test_dashboard_chat_transient_stream.py and test/test_kas_manual_compact.py as co-tenants (53 passed each), and one shuffled order, seed 27034 (16 passed).

Local gates: isort, flake8, black (changed files), mypy (src/kiro_crew/, no issues), the backend static scripts from the profile, 42 related test files (2,063 passed), and 13 chat retry files on this head (337 passed). semgrep could not install on this aarch64 host; CI's SAST job runs it.

Manual verification

A stdio probe against kiro-cli 2.32.0 (kiro-cli acp --agent-engine v3 --auth-method cli, a throwaway session, deleted after): prompt 1 named codeword ALPHA, prompt 2 named BRAVO, then _kiro/checkpoint/revertMultiple on prompt 2's user_message_id_assigned id answered {"success": true, "affectedFiles": [], "totalFiles": 0}. Asked for every codeword, the model answered ALPHA only, and messages.jsonl held a tombstone row (checkpoint_revert). A second revert of the same id answered success: false ("not found"). A real provider throttle was not forced.

Related Issues

Refs #21876

Pattern harvest

Rule candidate: review-prompt
Pattern: a retry that re-sends a prompt into a stateful session leaves the failed attempt in that session's history unless it removes it first.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@iamwhatever
iamwhatever requested review from a team and KiroCrewCaptain as code owners October 18, 2026 13:29
@github-actions

github-actions Bot commented Oct 18, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 713f5c7087775e3cf1bff794cb0ea8d4841aeb60 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Advisory — does not block merge.

@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 18, 2026
@github-actions

github-actions Bot commented Oct 18, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 713f5c7087775e3cf1bff794cb0ea8d4841aeb60 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Check that the 6/12/24 s throttle wait should apply on non-KAS harnesses. Its only stated reason is KAS's own ~6 s retry inside each attempt.

[DESIGN-REVIEWED] 713f5c7

Description read

[DESCRIPTION-READ] 23a06ec53c165c8fb9600a4b35a6e686130cc7e7e822ec2e5469262c1a0ce1a9

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 18, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — 🟡 CONCERNS

Premise-level review of 713f5c7087775e3cf1bff794cb0ea8d4841aeb60 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: CONCERNS

The new throttle classifier repeats one the same ladder already runs, and turn_stats.retries is written but nothing in the repo reads it.

Not justified as shipped

  • Item 3 — duplicate of src/kiro_crew/taskq/adapters/acp_provider.py (classify_text, which maps PROVIDER_ERROR_THROTTLE to KIND_RATE_LIMITED). The ladder already reaches it through _shared_dependency_delay (chat_turn/recovery.py:324, 1 call site) on every transient error. Both read the same classify_provider_error, but the two disagree: a named quota exception counts as CONCURRENCY_EXCEEDED in the existing one and as a throttle in the new err.throttle.
    Subtraction: delete AcpError.throttle and acp_error_is_throttle, and set _throttled from the DependencySignal the ladder already computes (kind == KIND_RATE_LIMITED).
    Clears when: the deletion is made, or a test shows a throttle frame that the dependency signal misclassifies.
  • Item 4 — zero consumers: a grep for "retries" and turn_stats.*retries outside tests finds only the writer at turn_stats.py:141. The description says "the footer does not render retries", and no latency report in the repo reads turn_stats. The recovery-outcome counter (item 5) already records the count per mechanism.
    Subtraction: drop the retries parameter and key from _attach_owned_turn_stats.
    Clears when: a counted in-repo reader of turn_stats.retries exists.

What this change ships

Inventory (7 items) — 5 justified

Intent: stop a throttled chat retry from piling copies of the prompt into KAS history, and make throttle retries visible. FIX.

  1. A throttled turn's re-send first removes the failed copy of the prompt from KAS history — justified
  2. Throttle retries wait 6/12/24 s instead of 2/4/8 s — justified
  3. Throttle detection tagged on the error and read by a new predicate — duplicate of taskq/adapters/acp_provider.py classify_text
  4. Landed row's turn_stats records a retry count — zero consumers
  5. Recovery-outcome counter gains throttle/transient_5xx — justified
  6. Provider-level discard_unanswered_prompt passthrough — justified
  7. Model-fallback re-send also removes the failed copy — justified

[FIRST-PRINCIPLES-REVIEWED] 713f5c7

Description read

[DESCRIPTION-READ] d56b4b3ca0e348a4ca0b9281e3582ca0ea9a3f9f9fe23c5e9a2844d20e3f5755

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • NOT JUSTIFIED AS SHIPPED: turn_stats.retries has no in-repo reader span=6abbfa1da1a4

rebutted: the frozen Goal asks for this record ("records how many retries the turn took"), and #21876 asks for it in turn_stats.
Reader: the R3 latency analyzer (research/crewmate-perf/r3/analyze.py, outside this repo) reads meta.turn_stats off each slot's history file (ttft_ms, elapsed_ms); retries rides the same row and the same read.
Precedent on main: ttft_ms has this exact shape today: written to meta.turn_stats, not rendered by the footer, no in-repo reader, kept so it is readable without telemetry on.
Telemetry carries the aggregate (kirocrew.watchdog.recovery.outcome, per-minute, no slot key); only the per-turn field lets one turn's model time split into first attempt and retries.
Cost: one int, written only on a retried top-level turn (omitted at 0).

@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • Confirm _turn_emitted is set by every tool call KAS runs before a rewind

confirmed: both rewind sites sit on arms gated not _turn_emitted, and _run_chat sets it on every EVENT_TOOL_CALL (dashboard/chat_runner.py, "tool side effect — transient retry now unsafe").
KAS sends a tool_call session/update before it runs any tool (approval keys on that frame's toolCallId); a KAS sub-agent's child tool frames fall through _handle_kas_subagent to the shared parser and become EVENT_TOOL_CALL too.
revertMultiple restores files only for write-type tool_call rows in the hidden range, so any such row means the runner saw the frame and took the no-rewind arm.
A throttle after a tool round takes the post-token continuation, which never rewinds (test_a_throttle_after_partial_output_keeps_the_attempt); the live probe's revert answered affectedFiles: [].

@github-actions

github-actions Bot commented Oct 18, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 713f5c7087775e3cf1bff794cb0ea8d4841aeb60 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 713f5c7

Verdict parsed from the review's SHA-scoped output markers for commit 713f5c7087775e3cf1bff794cb0ea8d4841aeb60.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 713f5c7087775e3cf1bff794cb0ea8d4841aeb60: <one-sentence reason>

@github-actions

github-actions Bot commented Oct 18, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 713f5c7087775e3cf1bff794cb0ea8d4841aeb60 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 713f5c7

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 713f5c7087775e3cf1bff794cb0ea8d4841aeb60: <one-sentence reason>

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not ready. Two repo invariant tests fail on this head, so readiness cannot pass.

Head reviewed: e7779a7.

What I checked:

  • The rewind gate in discard_unanswered_prompt: one shot, only when the turn is done, only when no later prompt started (_prompt_starts ordinal), cleared by a steer, and only with the advertised checkpoints capability. Both call sites sit on the not _turn_emitted arms and run before the Stop re-read, so a Stop during the rewind still drops the re-send.
  • _turn_retries lifecycle: reset at the start of a top-level non-recovery turn, read into turn_stats before the landed-turn reset, emitted as recovered on an ok turn and as exhausted on the terminal no-requeue arm.
  • The new tests patch the backoff (_recovery_delay, transient_retry_delay) and use tmp_path, so they do not sleep for the 6 s curve.
  • The design and first-principles dispositions on this head. I agree with both answers.

Failing on this head (from the CI logs):

  • test/test_acp_bindings.py::test_the_member_set_is_the_frozen_inventory (Linux shard 5, Windows shard 5): assert ['discard_unanswered_prompt'] == [].
  • test/test_harness_parity.py::test_every_runtime_path_identity_test_is_declared (Windows shard 6): session_handle.py:4051 in discard_unanswered_prompt() compares ACP_BACKEND_KAS.

Inline comments give a fix for each. No other findings.

Comment thread src/kiro_crew/acp/session_handle.py Resolved
Comment thread src/kiro_crew/providers/acp.py Resolved
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 18, 2026
@iamwhatever
iamwhatever force-pushed the fix/throttle-retry-no-stacked-prompt branch from e7779a7 to 350864c Compare October 18, 2026 14:33
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 18, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • NOT JUSTIFIED AS SHIPPED: turn_stats.retries has zero in-repo readers span=ca0136b22af1

rebutted: the field is a Goal deliverable ("records how many retries the turn took"), and #21876 asks for the count in turn_stats. Deleting it drops the Goal; an in-repo footer reader is a UI change, which "Not a goal" rules out.
Reader: the R3 latency analyzer (research/crewmate-perf/r3/analyze.py, outside this repo) reads meta.turn_stats off each slot's history file; retries rides the same row and the same read as ttft_ms and elapsed_ms.
Precedent on main: ttft_ms has this exact shape. turn_stats.py writes it to meta.turn_stats, the footer does not render it, and nothing in src/, website/src or scripts/ reads it.
The recovery-outcome counter is a per-minute aggregate with no slot or turn key, so it cannot split one turn's model time into first attempt and retries; the per-turn field can.
Cost: one int, written only on a retried top-level turn (omitted at 0). Same rebuttal as on e7779a7; this head only fixed the two red invariant tests.

@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • Check that user_message_id_assigned reaches the client before a real throttle error

confirmed: KAS 2.32.0 sends the row id before it starts the model request. In its session/prompt handler, one step appends the user row to the message store and then awaits connection.sessionUpdate({kind:"user_message_id_assigned", userMessageId}). That step runs before triggerSessionExecution, so before any model call.
A throttle comes from that execution (waitForCompletion, including KAS's own 3-try retry); its catch rethrows it as the session/prompt error response. The pre-execution failure path (model settle, required MCP servers) also stores the row and sends the id before it throws.
Only agentInitiated prompts skip the id. Crew never sets agentInitiated (no match in src/).
Crew gets both frames from one ordered session queue: the runtime reader puts notifications and the prompt response there in stdout order, and AcpSessionHandle.prompt checks is_response_for(req_id) per frame. So the row is recorded before the error raises. test_kas_unanswered_prompt_rewind.py drives the real prompt() loop in that order (id frame, then a ClientThrottleError response) and asserts revertMultiple for that row.
R3 §2.6's live throttles left the failed rows in history (four copies of P2), and the step that stores a row is the step that sends its id. A forced live throttle was not run: it needs real 429s from the service.

…story

When the provider throttles a chat turn before any output, KAS has already
stored the user row, so the dashboard's verbatim re-send left one copy of the
message per attempt in the model's history. The session handle now keeps the
row id KAS reports for each prompt (user_message_id_assigned), and before each
re-send of a throttled turn the runner asks the provider to rewind it with
_kiro/checkpoint/revertMultiple. Nothing is sent off KAS, without the
advertised checkpoints capability, after a later turn or a steer, or twice.

A throttle (AcpError.throttle, tagged from the raw frame) waits on its own
curve from 6 s, about one KAS attempt, instead of 2 s. Re-sends are counted
per mechanism on the slot: the landed row's turn_stats records retries, and
kirocrew.watchdog.recovery.outcome gains throttle and transient_5xx points.

Refs #21876
@iamwhatever

Copy link
Copy Markdown
Contributor Author

Backend Tests (Windows) (6) failed on 713f5c7087 with a known flake, not this change: test/test_webex_dispatch_queue.py::TestWebexSharesTheQueueWithOtherTransports::test_a_foreign_entry_is_never_answered_and_never_consumed hit the pytest timeout and crashed worker gw3 (job log).

@bolichen97
bolichen97 dismissed their stale review October 18, 2026 16:03

Both earlier findings are fixed: discard_unanswered_prompt is now in the frozen _MEMBERS inventory, and the backend identity check no longer needs a declared test entry. Head 713f5c7 is the same patch rebased onto main. Approval follows once readiness passes.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 18, 2026

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both earlier findings are fixed on this head and readiness passed. Approving.

@bolichen97
bolichen97 merged commit a9898a0 into main Oct 18, 2026
152 of 153 checks passed
@bolichen97
bolichen97 deleted the fix/throttle-retry-no-stacked-prompt branch October 18, 2026 16:30
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 18, 2026
KiroCrewCaptain pushed a commit that referenced this pull request Oct 19, 2026
…ext turn (#21937)

Follow-up to #21907.

A Stop that ended a throttle episode reset the transient budget (or, for
a Stop still resolving, nothing) but kept _turn_retries. A later
Continue runs as a recovery turn, which keeps the running count, so it
reported the stopped episode's retries in turn_stats and emitted a
"recovered" outcome for it. Each of the four Stop arms on the ladder
(a Stop still resolving, before the backoff, after it, and during the
model-fallback wait) now clears the count.

Refs #21876

Co-authored-by: Joe Guo <zejiangg@amazon.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants