Skip to content

fix(acp): classify kiro-cli's "failed to process the request" as transient - #11875

Merged
bolichen97 merged 1 commit into
kirodotdev:mainfrom
jeeshofone:fix/transient-failed-to-process-request
Sep 24, 2026
Merged

bolichen97 merged 1 commit into
kirodotdev:mainfrom
jeeshofone:fix/transient-failed-to-process-request

Conversation

@jeeshofone

@jeeshofone jeeshofone commented Sep 18, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

When the model backend returns kiro-cli's wrapper The service failed to process the request (request_id: ...), the dashboard chat ends the turn with a terminal error card and no retry. Sibling transient 5xx failures recover automatically, so this wrapper makes people click Continue and stops unattended work.

Why it matters

This wrapper reports a momentary backend failure. Retrying is useful before any output or tool work occurs, but replaying the original prompt after a tool mutation can execute that mutation twice.

What changed (motivation → approach → change)

src/kiro_crew/acp/client.py adds _RE_PROCESS_FAILED beside the existing generate-failed and 5xx classifiers. It is scoped to structured error data, preserves auth and malformed-request precedence, reports ProviderErrorClass.RETRYABLE, and formats the error with its request_id and retry guidance. src/kiro_crew/llm_helpers.py adds the lower-case marker so raw and formatted forms use the same retry decision.

Unattended same-model and fallback-chain retries replay the original prompt only before any assistant text or tool call has occurred across all attempts. One sticky tool-activity flag now gates both retry paths. A textless completed tool call makes a later transient error terminal instead of replaying the mutation. Prompt-busy keeps its existing retry contract, but tool activity from that attempt remains sticky if a later attempt hits a transient error. docs/system-specs/modules/providers.md records this boundary.

Backwards compatibility

Compatible: one additional wrapper is retryable before output or tool activity. Existing auth, validation, partial-output, prompt-busy, and fallback verdicts stay unchanged. A completed tool call intentionally prevents original-prompt replay.

Tests

test/test_acp_client.py covers raw classification, provider classification, formatter wording, request_id retention, structured-data scoping, and auth precedence.

test/test_llm_helpers.py proves a transient error before tool activity still retries, a textless mutation blocks same-model and fallback replays, and activity remains sticky across a prompt-busy attempt. The complete nine-file ACP/LLM set passes 1,042 tests. Both no-replay regressions fail at pytest call phase on reviewed head 97cdbbfb4. Full mypy passes 1,706 files, and all Black, isort, flake8, history, brand, subprocess, Agent SDK, async, changelog, docs, diff, body, and single-commit gates pass.

Manual verification

N/A — unit coverage drives the raw classifier, formatter, retry loop, prompt-busy transition, and fallback chain directly without performing real tool I/O.

Screenshots / video

N/A — no user-visible UI changes.

Related Issues

Closes #11873

Pattern harvest

Rule candidate: review-prompt
Pattern: an original-prompt retry must require both no emitted text and no fired tool call across every prior attempt.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@jeeshofone
jeeshofone requested review from a team and buluoray September 18, 2026 19:41
@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: checking Automated validation is still running labels Sep 18, 2026
@github-actions

github-actions Bot commented Sep 18, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5, fork) — 🟡 CONCERNS

Design-level review of 62052b1e74d84937607d77829e96c7320beb8d7c via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: CONCERNS

Unattended turns that fired any tool now die on the first transient 5xx — and a higher-layer whole-turn retry would re-run the mutation anyway.

Watch

The guard extension ("Any fired tool call makes a later transient error terminal on that path") removes same-model transient resilience for essentially every agentic unattended turn — workflow agents (workflows/agent_exec.py, agent_pool.py) fire a tool early, so a momentary blip that yesterday retried up to 3 times now fails the turn. That is the right call at this layer only if no caller reacts to the terminal error by replaying the whole turn; if one does, double-execution moves up a layer and the guard bought fragility without safety.
Clears when: the author confirms no unattended caller (workflows, taskrunner, cron) auto-replays a turn that fails with this now-terminal error.

Suggestions

As a follow-up, port the interactive path's continue-instruction recovery (re-queue "continue" on the same live session, not the original prompt) into stream_and_collect — it restores post-tool transient resilience without the replay hazard this PR closes.

[DESIGN-REVIEWED] 62052b1

@github-actions

github-actions Bot commented Sep 18, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5, fork) — 🟡 CONCERNS

Premise-level review of 62052b1e74d84937607d77829e96c7320beb8d7c via the fork AI-review pipeline — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: CONCERNS

The PR's own rule ships half-applied: the model-rejected replay (Case 2.5) still re-runs the original prompt after a fired tool call.

Not justified as shipped

  • Item 3 — symptom-level (point patch): the description's harvested rule — "an original-prompt retry must require both no emitted text and no fired tool call across every prior attempt" — gates Case 2 and Case 2.75 but leaves 1 counted sibling in the same function: Case 2.5's model-rejected replay (src/kiro_crew/llm_helpers.py:2476) is guarded by not result_text only, so a prompt-busy retry that fired a tool followed by a model rejection still replays the mutation (grepped the four retrying = True original-prompt branches; prompt-busy is the one the description declares retained).
    Clears when: Case 2.5 is gated by the same _turn_tool_activity flag, or the description/spec states Case 2.5 is deliberately left and why.

What this change ships

Inventory (6 items) — 5 justified

Intent: FIX — make kiro-cli's "failed to process the request" wrapper auto-retry like its sibling transient errors instead of ending the turn (Closes #11873).

  1. That wrapper now retries automatically instead of showing a terminal error card — justified
  2. Its error card shows retry guidance and keeps the request_id — justified
  3. Unattended same-model retry now stops once any attempt fired a tool call — symptom-level: sibling Case 2.5 replay left ungated
  4. Tool activity from a prompt-busy attempt stays sticky for later transient errors — justified
  5. Raw and formatted wrapper text classify as transient in the string-fallback classifier — justified
  6. providers.md records the replay boundary — justified

Item 3 reverses the base pin ("pre-existing semantics, deliberately untouched"), but the pin's own message names the double-mutation harm and distinguishes Case 2 only by replay count, and the description declares the reversal with base-failing regressions — supported, not silent.

[FIRST-PRINCIPLES-REVIEWED] 62052b1

@github-actions

github-actions Bot commented Sep 18, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5 Review (fork) — ✅ no blocking findings

Reviewed 62052b1e74d84937607d77829e96c7320beb8d7c via the fork AI-review pipeline; updated in place on each push.

Review details

Advisory only — the extended tool-activity block turns a momentary backend blip into a hard failure for unattended tool-using turns.

FINDING — src/kiro_crew/llm_helpers.py:2376 — and not _turn_tool_activity gates Case 2's same-model retry on a flag set by the tool-call announcement (EVENT_TOOL_CALL is emitted from the tool_call session update, which precedes session/request_permission — see acp/client.py:5189 — so a gate-denied or REJECT_ALL tool that never executed sets it too), so an unattended turn that touched any tool and then takes a pre-text transient 5xx now loses all three retries with nothing compensating: _cron_stream_with_posttoken_resume re-raises when no text streamed (slack/gateway.py:1552, whose own docstring states "that window is stream_and_collect's own retry's job"), the outer cron retry is gated on not _prompt_dispatched (slack/gateway.py:5980, set at 5593 before the stream), so the cycle lands on record_failure() and five such blips reach _AUTO_PAUSE_THRESHOLD and durably disable the job → Fix: gate Case 2 on evidence a tool actually EXECUTED (a completion-bearing tool event / executed_calls) rather than the pending announcement, or re-queue a continue instruction as the subagent ladder does — both need code outside the changed lines, so this cannot gate the merge.

[OPUS-REVIEWED] 62052b1

@github-actions

github-actions Bot commented Sep 18, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review (fork) — ✅ no blocking findings

Reviewed 62052b1e74d84937607d77829e96c7320beb8d7c via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 62052b1

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 18, 2026
@jeeshofone

Copy link
Copy Markdown
Contributor Author

Disposition for 2cdca79f9.

F1 (fenced, BLOCKING): transient replay can duplicate completed tool mutations. The adjudicator flagged it as an edge case that cannot occur. The same surfaces on this branch confirm that. The new pattern only marks the error transient; it enters the retry ladders _RE_GENERATE_FAILED already uses, and every one of them decides between replay and continuation on activity, not on the error text:

  • Dashboard, pre-token: chat_runner.py:14947 and :15075 gate verbatim replay on not _turn_emitted.
  • Dashboard, post-token: chat_runner.py:15173 runs only when _turn_emitted is true and re-prompts the same live session with a CONTINUE instruction, so a completed tool result is kept and never re-run.
  • Compaction sibling: chat_runner.py:13237, same not _turn_emitted guard.
  • Subagents: subagent_manager/run.py:1449 computes _had_activity = bool(result_text) or turns > 0 or info.tool_count > 0; :1466 and :1561 send _TRANSIENT_CONTINUE_MSG instead of the original prompt once there was activity.

A tool that ran before the wrapper arrived therefore lands in the post-token or _had_activity branch and gets a continuation. Recording the override is a maintainer action. This comment provides the verification the lane requires.

Backend Tests (Windows) (2) failed test_ci_fleet_routing_expression_parity.py::test_every_ci_yml_resolver_consumer_reads_the_resolver_not_a_literal, which is red on main since #11699 (d0ff8107d); #11876 carries the census update. Nothing in this diff touches .github.

@jeeshofone
jeeshofone force-pushed the fix/transient-failed-to-process-request branch from 2cdca79 to 145f694 Compare September 18, 2026 22:32
@jeeshofone

Copy link
Copy Markdown
Contributor Author

Board note: rebased onto main with #11876, which fixes the resolver-consumer parity test behind (3.12, 2) and Windows (2). Same diff, new head 145f694e4. The fenced F1 disposition above stands for this head; the code at the finding site is unchanged.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 18, 2026
@jeeshofone
jeeshofone force-pushed the fix/transient-failed-to-process-request branch from 145f694 to 4019c28 Compare September 18, 2026 23:08
@jeeshofone

Copy link
Copy Markdown
Contributor Author

Board note: both review lanes reported "review incomplete" on 145f694e4 (lane jobs failed about two minutes in, same minute as on #11888, right after the rebase); CI was otherwise green. Re-pushed the same tree as 4019c285d so the lanes run again. The fenced F1 disposition above stands; the code at the finding site is unchanged.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 18, 2026
@jeeshofone

Copy link
Copy Markdown
Contributor Author

Board note: the two lane failures on this head are a repo-wide outage, not a verdict. Every fork lane run since about 22:42Z ends review incomplete after Connection failed: error sending request to the model backend, across six PRs in the last ten minutes alone. Filed as #11911. No further pushes from my side until the lanes are reachable; a lane re-run on this head is the clearance.

@jeeshofone
jeeshofone force-pushed the fix/transient-failed-to-process-request branch from 4019c28 to 5d362b1 Compare September 19, 2026 17:38
@jeeshofone

Copy link
Copy Markdown
Contributor Author

Re-pushed as 4019c285d -> 5d362b126 with the tree unchanged (same-tree amend), now that #11976 has widened the fork lanes' egress allowlist on main. This head runs the review lanes against the fixed workflow; the earlier "incomplete" verdicts were the #11911 egress block, not findings.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 19, 2026
@jeeshofone

Copy link
Copy Markdown
Contributor Author

Rebased locally onto current origin/main to refresh the failed readiness state. The PR patch is byte-identical across the rebase (range-diff =). Exact changed tests: 632 passed. Formatting, isort, flake8, mypy, comment-history, brand-name, SDK-boundary, and zero-.github-diff gates pass. New head to follow: b9b874ee3a58d1f2f3658611170db33445017897. The remaining fork-wide Opus workflow defect is tracked in #12099.

@jeeshofone
jeeshofone force-pushed the fix/transient-failed-to-process-request branch from 5d362b1 to b9b874e Compare September 21, 2026 01:35
@jeeshofone

Copy link
Copy Markdown
Contributor Author

All four fork workflows on this fresh head are still waiting for the maintainer approval click and none has started. Please approve the workflow run so the new head can receive a current review/readiness verdict.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 21, 2026
@jeeshofone

Copy link
Copy Markdown
Contributor Author

Windows shard 1 failed the known unrelated risk-annotation timing test with the exact #12376 signature (test_a_trusted_tool_call_is_flagged_on_its_card_and_takes_a_verdict: card missing decisions_tool_risk). I added this second recurrence and job evidence to #12376. The PR diff cannot reach that subsystem; its exact 632 ACP tests passed after the rebase. I am keeping this branch scoped. The current required readiness blocker remains the shared fork Opus failure in #12099.

@jeeshofone
jeeshofone force-pushed the fix/transient-failed-to-process-request branch from b9b874e to 62052b1 Compare September 24, 2026 04:05
@jeeshofone

Copy link
Copy Markdown
Contributor Author

Fresh exact head 62052b1e74d84937607d77829e96c7320beb8d7c is rebased onto current main and locally green: 1,042 focused ACP/LLM tests, full mypy (1,706 files), all static/docs/body guards, and exact-head GPT + Opus local reviews pass. Please approve the fork workflows for this head when convenient.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 24, 2026
@bolichen97
bolichen97 enabled auto-merge (squash) September 24, 2026 06:53
@bolichen97
bolichen97 merged commit 30dc1f0 into kirodotdev:main Sep 24, 2026
96 checks passed
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Sep 24, 2026
wknight1 pushed a commit to wknight1/KiroCrew that referenced this pull request Oct 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Transient classifier misses kiro-cli's "The service failed to process the request", so the turn dies with no retry

3 participants