Repository navigation
fix: keep an unsure Jev tier off a smaller context window - #13167
Conversation
Design Review (Fable 5) — 🟡 CONCERNSDesign-level review of Design-Verdict: CONCERNS PR description documents a superseded refusal design ("lands one gear up on medium", Watch
[DESIGN-REVIEWED] 17c8404 |
First Principles Review (Fable 5) — 🟡 CONCERNSPremise-level review of First-Principles-Verdict: CONCERNS The description promises a refused tier "lands one gear up" via Not justified as shipped
What this change shipsInventory (10 items) — 7 justifiedIntent: keep a low-confidence or ill-fitting tier answer from moving a chat onto a model with a smaller context window, whose compaction is irrecoverable — a FIX (enforces the base spec's own pin, "routing never silently lands on a smaller window").
[FIRST-PRINCIPLES-REVIEWED] 17c8404 |
GPT 5.6 Review — ✅ human override acceptedHuman judgment by @iamwhatever overrides the GPT 5.6 finding for [GPT-OVERRIDE] 17c8404 This comment is updated in place on each push. The model was not re-run because an authorized human decision supersedes it. False positive or not applicable? A repository writer can comment: |
Opus 5 Review — ✅ no blocking findingsReviewed Review detailsA post-switch rollback that raises records FINDING — src/kiro_crew/dashboard/chat_runner.py:4835 — when the post-switch window re-check sets [OPUS-REVIEWED] 17c8404 Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
b4e8d86 to
5e2b77f
Compare
5e2b77f to
1068527
Compare
ebb7798 to
563e03a
Compare
563e03a to
79e6880
Compare
79e6880 to
41c460a
Compare
41c460a to
40cbeaf
Compare
model.route applied a tier on the tier alone, so a half-sure `simple` moved a session onto a model with a fifth of the room. The context meter is a percentage of the SERVED window, so a transcript sitting well inside a large one can cross the session's compaction threshold the moment the window shrinks, and that turn ends by handing the backend a history it replaces with a summary no later switch back recovers. The spec even carried a guarantee it only partly kept: "routing never silently lands on a smaller window" was true of an id's spelling, not of the window a correctly-spelled pin names. Three rules now stand between the tier's model and the seam that applies it, and each applies ONLY to a move to a SMALLER window. They are scoped by `is_smaller_window`, the single owner of that question, so they cannot come to govern different sets of moves, and an unknown window on either side refuses nothing. The ROLLBACK rule is asked first, because it is a question about the seam rather than about the turn: a shrink is only ever applied because it can be TAKEN BACK if what actually serves turns out smaller still, and the model to go back to is the one the session was already on. A session the backend never named a model for reads that as "" -- there is no id to ask for -- so a smaller window is refused on exactly the grounds an unknown one is: nothing here confirms the turn could be put back. A window at least as large needs no way back, so an upgrade pays nothing for it. The CONFIDENCE rule is the point's own floor. MIN_DOWNGRADE_P is 0.80, a plain module constant for memory.recall's KEEP_THRESHOLD reason -- it is the meaning of the answer rather than a knob. The FIT rule refuses when the turn's own reading would land at or above this session's compaction threshold in the target window. Its estimate no longer reads token-dense text as a quarter of its size: CHARS_PER_TOKEN answers for LATIN text, and CJK, Indic or symbol-dense input tokenizes several times denser, so only ASCII is divided and every other character counts as a token of its own. That is the conservative direction and the only safe one here -- it can refuse a downgrade the prompt would have fitted, which costs one turn on a larger window, while the opposite costs a history nothing recovers. A refused tier STAYS PUT. The turn keeps the window it already has, which is a window these rules read and accepted, and nothing is asked of the provider at all: there is no switch to undo, nowhere to land, and no turn to stop. The row carries the suppressed tier and its p under error smaller-window-refused, written where the rules answer and behind the same premise predicate the apply path takes, because rows are durable and never rewritten and one appended for a superseded answer would outlive a decision nothing acted on. The CURRENT side is the window the session is served on whenever the provider reports one, and the registry only when it does not: that reading is the meter's own denominator, so a registry entry lagging it would hide a shrink the meter can already see. Both rules size the id that was ASKED for, so a backend answering a tier-policy substitution is asked once more after the switch, against the served window, and a turn that landed on a window the rules did not authorise is put back on the one it had -- which the ROLLBACK rule is what makes expressible. A substitution under an UPGRADE can still reach that check with nothing named to go back to, and then the turn keeps the model it is on: the residual any switch that cannot be undone has always had. A raise from either switch takes the paths a failed set_model already took. Both rules sit on the routing path and NOT in AcpClient.set_model: an owner picking a smaller window by hand is the answer to the question this point asks, and a check there would refuse the pick as well. The prompt gains one sentence saying which direction an error costs more. It names no model and no price, the rule the tier descriptions already follow.
40cbeaf to
17c8404
Compare
|
/ai-review override gpt 17c8404: design A accepts this residual by choice — the abort path it asks for was removed after producing four one-shot-state defects; a downgrade now requires a rollback target, so this site is reachable only by backend substitution, which pre-PR code streamed on silently without any recheck |
Human judgment recorded@iamwhatever marked the gpt AI finding as false positive, not applicable, or explicitly accepted for
This decision applies only to this commit. A new push requires a new judgment. |
buluoray
left a comment
There was a problem hiding this comment.
Reviewed at 17c84048. No hardcoded model id in source (the two haiku ids are test fixtures). _jev_downgrade_refused runs before set_model and refuses a smaller window on missing/low p (< 0.80), unknown usage, an unreadable threshold, an unnamed current model, or usage+prompt reaching the smaller window's compaction threshold; after a switch it re-checks the served window and switches back. full_message is only used locally for token estimation; the only new text sent to routing is the fixed TIER_ASYMMETRY. Nine mutations of the guards each redden the tests.
Non-blocking: in the post-switch branch the comment says a switch-back that raises "is still reported as the refusal it is", but the except records ERROR_SWITCH_FAILED and the turn stays on the smaller window; align the comment or the category.
…#13167) model.route applied a tier on the tier alone, so a half-sure `simple` moved a session onto a model with a fifth of the room. The context meter is a percentage of the SERVED window, so a transcript sitting well inside a large one can cross the session's compaction threshold the moment the window shrinks, and that turn ends by handing the backend a history it replaces with a summary no later switch back recovers. The spec even carried a guarantee it only partly kept: "routing never silently lands on a smaller window" was true of an id's spelling, not of the window a correctly-spelled pin names. Three rules now stand between the tier's model and the seam that applies it, and each applies ONLY to a move to a SMALLER window. They are scoped by `is_smaller_window`, the single owner of that question, so they cannot come to govern different sets of moves, and an unknown window on either side refuses nothing. The ROLLBACK rule is asked first, because it is a question about the seam rather than about the turn: a shrink is only ever applied because it can be TAKEN BACK if what actually serves turns out smaller still, and the model to go back to is the one the session was already on. A session the backend never named a model for reads that as "" -- there is no id to ask for -- so a smaller window is refused on exactly the grounds an unknown one is: nothing here confirms the turn could be put back. A window at least as large needs no way back, so an upgrade pays nothing for it. The CONFIDENCE rule is the point's own floor. MIN_DOWNGRADE_P is 0.80, a plain module constant for memory.recall's KEEP_THRESHOLD reason -- it is the meaning of the answer rather than a knob. The FIT rule refuses when the turn's own reading would land at or above this session's compaction threshold in the target window. Its estimate no longer reads token-dense text as a quarter of its size: CHARS_PER_TOKEN answers for LATIN text, and CJK, Indic or symbol-dense input tokenizes several times denser, so only ASCII is divided and every other character counts as a token of its own. That is the conservative direction and the only safe one here -- it can refuse a downgrade the prompt would have fitted, which costs one turn on a larger window, while the opposite costs a history nothing recovers. A refused tier STAYS PUT. The turn keeps the window it already has, which is a window these rules read and accepted, and nothing is asked of the provider at all: there is no switch to undo, nowhere to land, and no turn to stop. The row carries the suppressed tier and its p under error smaller-window-refused, written where the rules answer and behind the same premise predicate the apply path takes, because rows are durable and never rewritten and one appended for a superseded answer would outlive a decision nothing acted on. The CURRENT side is the window the session is served on whenever the provider reports one, and the registry only when it does not: that reading is the meter's own denominator, so a registry entry lagging it would hide a shrink the meter can already see. Both rules size the id that was ASKED for, so a backend answering a tier-policy substitution is asked once more after the switch, against the served window, and a turn that landed on a window the rules did not authorise is put back on the one it had -- which the ROLLBACK rule is what makes expressible. A substitution under an UPGRADE can still reach that check with nothing named to go back to, and then the turn keeps the model it is on: the residual any switch that cannot be undone has always had. A raise from either switch takes the paths a failed set_model already took. Both rules sit on the routing path and NOT in AcpClient.set_model: an owner picking a smaller window by hand is the answer to the question this point asks, and a check there would refuse the pick as well. The prompt gains one sentence saying which direction an error costs more. It names no model and no price, the rule the tier descriptions already follow. Co-authored-by: Joe Guo <zejiangg@amazon.com>
Supersedes #12913. That PR was based on
fix/jev-model-route-ratchet, which merged as #12872 and was deleted, so GitHub auto-closed it and refuses to reopen it. Same head branch, same diff, rebased ontomain: the ratchet commit is already inmain, so what is left here is only the window guard. The refused-downgrade landing reads the ratchet's_jev_route_baselinelatch through its own_restore_to.The defect
model.routeapplied a tier on the tier alone. A half-suresimplemoved a session onto a model with a fifth of the room, and the context meter is a percentage of the served window: a transcript sitting well inside a large window can cross the session's compaction threshold the moment the window shrinks. That turn ends by handing the backend a history it replaces with a summary, and no later switch back recovers it. The spec even carried a guarantee it only partly kept — "routing never silently lands on a smaller window" was true of an id's spelling, not of the window a correctly-spelled pin names.The two rules
Both live in
chat_runner._jev_downgrade_refused, after the target is computed and beforeset_modelis resolved, and both are scoped bymodel_route.is_smaller_window— the single owner of "is this a shrink", so the two rules cannot come to govern different sets of moves.model_route.permits_smaller_window(p, current=, target=)— a smaller window needs the answer's own probability atMIN_DOWNGRADE_P(0.80) or above. An absent probability does not clear it.(used + prompt_tokens(message)) / target_window * 100 >= SessionManager.effective_autocompact_pct(key)— the reading the turn would END on once the window is the smaller one. The meter answers for the history the session already holds, so the prompt this turn SENDS is added at theCHARS_PER_TOKENdivisor the sibling points estimate with —full_message, the person's words under whatever request prefix, replayed history and hook context the turn carries, and deliberately not the excerpt the tier was classified from: without it a large message over a small history clears the test and compacts at the end of that same turn, which is the case the rule exists for. The rule PERMITS a shrink only on a reading that CONFIRMS the target holds this turn. A meter the provider itself calls unknown (context_usage_unknown, True exactly when a 0 means "not measured" rather than "empty" — the post-compaction and resumed state), a meter that cannot be reached, and an unreadable threshold all refuse. That costs one gear and nothing else, because a window that does not shrink never compacts, and it is not a veto on routing but only on shrinking THIS turn. A 0 the provider vouches for is a known small history, so a fresh session downgrades as before, with the prompt weighed beside it.The current side is the window the session is served on (
client.context_window_tokens()) whenever the provider reports one, and the registry only when it does not: that reading is the denominator the meter's own percentage is taken against, so a registry entry lagging it would hide a shrink the meter can already see. It is read ONCE (_jev_current_window) and the landing below is sized by that same number — a landing measured against the other reading can be handed the very window these rules refused, and the row would then say refused for a shrink that happened.An unknown window on either side refuses nothing (
known_windowgates onmodel_registry.has_known_window, becausemodel_windowanswers a guessed reference for an id it does not list, and vetoing on a guess pins routing to the session's own model for every unlisted one). Because that answer refuses nothing, the lookup asks the registry both as the id is WRITTEN and case-folded: both registry lookups are spelling-sensitive whileresolve_modelaccepts a pin on a lossless fold and answers the spelling the owner wrote, soClaude-Haiku-4.5asked only as written reads as unknown and carries the shrink..and-stay unfolded — the registry keeps those entries distinct and they are two different windows. A zero reading is "not measured yet" — the state a fresh session and a just-compacted one share — and also refuses nothing.A refused tier lands one gear up: the
mediumpin while its own window is not the one just refused, else the pre-route baseline, else the model is left alone. The landing is handed the same baseline value the unpinned restore gets (_restore_to), not the raw latch, because that value is already emptied while the model-fallback ladder holds the session: the latch there names the primary the ladder walked away from, and landing a refused tier on it would ask for the model that just failed.The refusal row carries the suppressed
tierand itspunder errorsmaller-window-refused, so which rule refused is readable from the row, and publishes no receipt. It is written at each exit that discards the tier rather than once where the rules answer: the locked re-pick guard drops the whole answer, and rows are durable and never rewritten, so one appended before that guard would outlive a decision nothing acted on. The two exits that return BEFORE those locks — no landing to ask for, and no switch seam at all — never reach that guard, so each asks the same premise itself through the one predicate that answers it (_jev_route_premise_moved: the arming flag, the model the turn started on, the shared pick epoch).Both rules size the id that was ASKED for, so the FIT rule is asked once more after the switch and before any prompt is streamed, against the window the session is now SERVED on. A config-option backend answering a tier-policy substitution leaves the turn on a different model whose id can still read as the requested one; only the meter says otherwise, because it is rebased to whatever serves. A turn that landed on a window the rules did not authorise is put back on the one it had, and the refusal is recorded like any other dropped tier.
Both rules are deliberately not in
AcpClient.set_model: an owner picking a smaller window by hand is the answer to the question this point asks, and a check there would refuse the pick too.Why 0.80
Read off
~/.kiro/crew/decisions/decisions-*.jsonl, n=17 answeredsimple:tool_risk.RISKY_CONFIDENCE_THRESHOLD) is the one taken rather than a second number.psimpleanswers: 9 are plans, judgement calls or multi-instruction turns.It is a plain module constant, not a config knob, for
memory.recall'sKEEP_THRESHOLDreason: it is the meaning of the answer, and a configurable floor is a second, undocumented way to turn the rule into a no-op.The rules are deliberately one-way. A symmetric floor would kill the
mediumtier outright — everymediumanswer in the sample is under 0.80 — and would push low-pcomplexanswers back toward a possibly smaller baseline, which is worse.Prompt
One sentence added beside the three tier descriptions: being wrong in the two directions does not cost the same, so
simpleis for a request that is self-contained and wholly visible. It names no model and no price, the rule the descriptions already follow.What was tested
python3 -m pytest test/test_decisions_model_route.py test/test_decisions_model_route_apply.py -q→ 116 passed. With the base PR's own ratchet file: 132 passed.One test per rule branch: pure function (both directions, the shrink owner on unknown windows, the floor inclusive at its own value plus
0.75 <= MIN_DOWNGRADE_P <= 0.85, a bool is not a probability, no config knob), landing (medium pin, medium pinned to the refused window, unpinned medium, nothing to land on), and the apply path driven through the real runner hook: refused → medium, refused → pre-route baseline over two turns, same window passes, unknown window passes, fit rule refuses at 150k/200k, fit rule passes at 10k, a pick landing during thedecideawait leaves no refusal row, a refused pin under an active fallback ladder never attempts the failed primary, a registry entry lagging the served window still refuses, the landing is sized by the same reading the rules refused on, a large prompt over a small history is refused, a prompt that alone will not fit is refused on a fresh session, a pin the owner cased differently is still sized, an unmeasurable meter refuses while a vouched-for zero does not, a substituted served model is put back while one that holds is left alone, and an empty landing writes no row once a pick has landed.Sixteen mutations, each caught, and each by the test that owns its branch:
test_a_pick_landing_during_the_await_leaves_no_refusal_rowtest_a_refused_pin_never_attempts_the_failed_primarytest_a_registry_entry_that_lags_the_served_window_still_refusestest_the_landing_is_sized_by_the_same_reading_the_rules_refused_ontest_the_rule_sizes_the_prompt_it_is_giventest_the_hook_is_handed_the_prompt_the_turn_sendstest_a_pin_the_owner_cased_differently_is_still_sizedtest_a_prompt_that_alone_will_not_fit_is_refused_on_a_fresh_sessiontest_an_unmeasurable_reading_refuses_and_a_vouched_for_zero_does_nottest_a_reading_that_cannot_be_reached_refusestest_a_substituted_served_model_is_put_backtest_a_substituted_served_model_is_put_backtest_an_empty_landing_writes_no_row_when_a_pick_landedGreen locally:
check_black_formatting.py(with the pinned 26.3.1),check_subprocess_encoding.py,isort,flake8,mypy src/kiro_crew(1699 files),check_comment_history.pygated at the base,check_brand_name.py,docs-lint.sh.Docs
docs/system-specs/modules/decisions.md§11 gains one paragraph: both rules, which shrink they are scoped by, where the current window is read, the landing order, which baseline the landing is handed, and where the refusal row is written. The §5 row-schema sentence gainssmaller-window-refused, and the over-broad guarantee is narrowed to what the id comparison actually promises.src/kiro_crew/docs/decisions.mdmirrors only the error-word sentence, which is the paragraph it shares.SessionManager.effective_autocompact_pctis the one line the fit rule cannot work without: the authoritative "override else global" resolution lives onCompactionCoordinator, andchat_runnerhas no public read of it and no precedent for reaching a privatesessions._attribute. It is the READ half of the existing publicset_autocompact_pct.model_registryis imported atmodel_route.py's module top rather than inside the function — this module is itself reached lazily from the turn path, so the statement costs nothing on any boot path, and the sibling helper's now-redundant local import of the same module is dropped with it. The point module is passed into_jev_downgrade_refusedinstead of re-imported there, because the wholedecisionspackage is reached lazily fromchat_runnerand a second import statement would be a second place holding that decision.Pattern harvest
Rule candidate: review-prompt
Pattern: an automated switch that moves a live session onto a smaller resource — a context window, a token budget, a timeout — applied from a classification alone, without reading the session's current consumption against the target's capacity. The defect class is not "the classifier was wrong"; it is that the applying seam never asked whether the target could hold what the session already has, so a correct-looking classification produces an irreversible loss (here: the backend replaces the history with a summary at the end of that same turn).
Nine checks a reviewer can apply to any such seam, all nine visible in this diff:
model.routewas the last of the three decision points still usingpfor logging only. The same question belongs on every point that applies an answer rather than displaying it.AcpClient.set_model) would also refuse a human picking that model by hand, which is the very answer the automatic path is guessing at. The guard belongs where the decision is made.Not a semgrep or lint candidate: each check needs the seam's semantics (which direction is the shrink, whose threshold applies, whether the reading is measured yet, which exits are terminal, which of two same-typed values is the sanitised one), and a syntactic rule would fire on every
set_modelcall site including the manual pick it must not touch.