Skip to content

fix(decisions): report a failed model.route put-back as the refusal it is - #20767

Merged
chenmingwei23 merged 1 commit into
mainfrom
fix/13588-rollback-failure-refusal
Oct 15, 2026
Merged

chenmingwei23 merged 1 commit into
mainfrom
fix/13588-rollback-failure-refusal

Conversation

@iamwhatever

@iamwhatever iamwhatever commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: When model.route puts a turn back after a too-small served window and that put-back fails, the log and the decision row say it was a refusal, not a failed switch.

The comment above the put-back in chat_runner._route_model_for_turn says a restore that raises "is still reported as the refusal it is rather than as a failed apply". The except Exception arm never read the _refused flag. So when the switch back raised, the warning blamed set_model(<tier model>), the switch that had worked, and the row was written as switch-failed with p dropped.

Why it matters

The smaller-window-refused row is how a reader sees the window rules doing their job (#13167). On this path it went missing, and the warning pointed at the wrong switch.

Not a goal

What changed (motivation → approach → change)

A failed put-back was logged as a failed switch. The error arm only asked whether a tier was chosen, never whether the rules had already refused it.

The error arm in src/kiro_crew/dashboard/chat_runner.py now checks _refused first. When it is set, it logs restoring <model> failed, writes the smaller-window-refused row with p, and returns. Every other failure keeps its switch-failed row and warning.

The row stays the plain refusal row on purpose. A refused tier never carries applied or model_used (spec §11), and a turn left on a substitute that cannot be put back already writes this same row on main. The warning is what names that the turn stayed on the model it switched to.

Backwards compatibility

Compatible: only the row and warning for a raising put-back change; every other path writes what it wrote before.

Removes nothing: no row kind, key, or path is taken away; one path writes the right existing row kind.

Tests

test/test_decisions_model_route_apply.py::TestTheWindowTheSwitchActuallyLandedOn::test_a_put_back_that_raises_is_still_the_refusal makes the backend substitute a smaller window, then makes the put-back raise. It checks one smaller-window-refused row with tier and p, no switch-failed row, and a warning naming the restore, not the forward switch.

Red on the parent (the row is switch-failed and the warning reads set_model('model-b') failed), green on this head. Determinism: 20/20 green on repeat, and green in a shuffled order beside its neighbours in TestTheWindowTheSwitchActuallyLandedOn, TestRefusalsKeepTheCurrentModel and TestARefusalStaysPut. It uses mocks and caplog only: no sleeps, clocks or ports.

Manual verification

N/A — unit coverage sufficient: the change is one branch in an error arm, and the test drives it through a real _run_chat turn.

Related Issues

Closes #13588

Pattern harvest

Rule candidate: review-prompt
Pattern: a flag set "so the handler reports X" that no except arm below it reads.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@iamwhatever
iamwhatever requested review from a team and KiroCrewCaptain as code owners October 15, 2026 09:45
@github-actions

github-actions Bot commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 58b0d4fb27676eef24775b4f38438348ebc3be63 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Advisory — does not block merge.

@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 15, 2026
@github-actions

github-actions Bot commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of 58b0d4fb27676eef24775b4f38438348ebc3be63 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

Check that the new warning ("the turn stays on the model it switched to") is still accurate when _sync_served_model, rather than set_model, is the call that raised.

What this change ships

Inventory (3 items) — 3 justified

Intent: when model.route refuses a smaller served window and the switch back raises, the log and the decision row should report a refusal, not a failed switch. This is a FIX (Closes #13588), and the new test fails on base.

  1. A failed put-back now writes a smaller-window-refused row with p, not switch-failed. — justified
  2. That warning now names the failed restore of the original model, not the forward switch that worked. — justified
  3. A regression test drives a real turn through the raising put-back. — justified

[FIRST-PRINCIPLES-REVIEWED] 58b0d4f

Description read

[DESCRIPTION-READ] a8f2343a36f8f27a55a358fc56ac711d751d10738e69eefe33123c92320f6f0e

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — 🟡 CONCERNS

Design-level review of 58b0d4fb27676eef24775b4f38438348ebc3be63 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: CONCERNS

When the put-back fails, the turn keeps running on the too-small window, but its durable row now looks the same as a refusal that did put the turn back.

Watch

  • The row hides that the put-back failed. The new arm calls _record_refused(), which writes only error=ERROR_WINDOW_REFUSED, p=…. Nothing in that row marks that set_model(_live_model) raised. The new warning says "the turn stays on the model it switched to", so the turn streams on the substituted smaller-window model, which is the outcome the window rules exist to stop. A later reader of the decision log sees an ordinary refusal and assumes the turn stayed put. Before this change, switch-failed was the wrong kind of row, but it did show that something failed. Now the only record of the failure is a log line.
    Clears when: the refusal row on this path carries something that says the restore did not take (for example the existing applied: False/model_usedpair, written the way thenot _chosen arm writes it), or the PR states why a log-only record of a turn left on a refused window is enough.

[DESIGN-REVIEWED] 58b0d4f

Description read

[DESCRIPTION-READ] 1d365b3365c3ec67c327b7bd29fe576b1e6190bbe154203a137da8e480b0400a

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 58b0d4fb27676eef24775b4f38438348ebc3be63 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 58b0d4f

Verdict parsed from the review's SHA-scoped output markers for commit 58b0d4fb27676eef24775b4f38438348ebc3be63.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 58b0d4fb27676eef24775b4f38438348ebc3be63: <one-sentence reason>

@github-actions

github-actions Bot commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 58b0d4fb27676eef24775b4f38438348ebc3be63 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 58b0d4f

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 58b0d4fb27676eef24775b4f38438348ebc3be63: <one-sentence reason>

@iamwhatever

iamwhatever commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor Author
  • span=c6c577a63dcf: The row now says the window rules held on a turn where they failed.

rebutted: smaller-window-refused records the rules' verdict on the tier, not which model served. The spec says so in docs/system-specs/modules/decisions.md §11: "A refused tier never carries applied or model_used either, because the model the turn moved to is the landing rather than the tier's", and "the row's error category is what describes the turn".
Same class, already on main: when a substitution has nothing to go back to (_live_model == "", chat_runner.py if _live_model: is skipped), the turn keeps the substitute and main already writes this same refusal row. The spec calls that "the residual any switch that cannot be undone has always had, and it is recorded rather than acted on". A raising put-back is that same residual.
Adding model_used/applied to refusal rows would change the row contract for every refusal and widen model_route.record_error. That is outside #13588, which asks only that this path stop writing switch-failed and dropping p.
The warning on this path already says the turn stays on the model it switched to, so the served model is not hidden from the operator log.

@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • Readers of smaller-window-refused on a turn that ran on the substitute

answered: the readers take the row as the rules' verdict on the tier, not as proof the turn stayed put. docs/system-specs/modules/decisions.md §11 says a refused tier carries neither applied nor model_used, and that a substitution which cannot be put back "is recorded rather than acted on". The _live_model == "" path on main already writes this row for a turn left on the substitute.
No reader keys the served model off this row. The operator warning on this path names that the turn stays on the model it switched to.

@iamwhatever
iamwhatever force-pushed the fix/13588-rollback-failure-refusal branch from 19fa9d2 to 58b0d4f Compare October 15, 2026 10:35
@iamwhatever

Copy link
Copy Markdown
Contributor Author

Rebased: main moved in src/kiro_crew/dashboard/chat_utils.py, src/kiro_crew/dashboard/handlers/agents.py and 7 test files since this head went green (b413800 -> 5b7b29f); CI re-running.

@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • span=31fa83dfef33: The row hides that the put-back failed.

rebutted (the second "Clears when" branch): a log-only record is enough, and the PR body now says why. smaller-window-refused records the rules' verdict on the tier, and spec §11 (docs/system-specs/modules/decisions.md) says a refused tier never carries applied or model_used.
Same class on main: when a substitution has nothing to go back to (_live_model == ""), the turn keeps the substitute and main writes this same row. The spec calls that residual "recorded rather than acted on". A raising put-back is that residual.
Adding applied/model_used to refusal rows changes the row contract for every refusal and widens model_route.record_error, which is outside #13588. The warning on this path names that the turn stayed on the model it switched to.

@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • Is the warning accurate if _sync_served_model is what raised?

answered: in practice it cannot raise. chat_turn/model_fallback.py:_sync_served_model getattr-reads slot.record_served_model and client.served_model, and DashboardState.record_served_model (dashboard/state.py) is a single attribute store. So on this arm the exception comes from set_model(_live_model), and the warning names that restore.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 15, 2026

@mbriones98 mbriones98 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Checks green and readiness passed at 58b0d4f; no behavior change.

@chenmingwei23
chenmingwei23 merged commit e188236 into main Oct 15, 2026
112 of 115 checks passed
@chenmingwei23
chenmingwei23 deleted the fix/13588-rollback-failure-refusal branch October 15, 2026 15:43
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

model.route: a failed rollback is logged as switch-failed, contradicting the comment above it

3 participants