Repository navigation
feat(decisions): let Jev route a chat turn to a model tier - #12267
Conversation
|
Intent: Let a chat session ask Jev how hard each message is and answer that message on the model mapped to that difficulty, so a small ask can run cheap and a hard one can run strong without the user driving the model picker between turns. |
Design Review (Fable 5) — 🟡 CONCERNSDesign-level review of Design-Verdict: CONCERNS Routing has no anchor model: after the first switch, every refusal, outage, or disarm keeps the last routed tier's model, not the owner's pick. WatchThe failure story claims a refusal "runs on the model the session was already using — what happens today for everyone," but that holds only for the first routed turn: [DESIGN-REVIEWED] 0893809 |
First Principles Review (Fable 5) — 🟡 CONCERNSPremise-level review of First-Principles-Verdict: CONCERNS Pin-vs-advertised matching is reimplemented in the new point beside the resolver the repo already names for exactly that job, and one response field ships unread. Not justified as shipped
What this change shipsInventory (10 items) — 8 justifiedIntent: let a chat owner hand each message's model choice to Jev by difficulty tier — an ADDITION.
[FIRST-PRINCIPLES-REVIEWED] 0893809 |
Opus 4.8 Review — ✅ no blocking findingsReviewed Review detailsNo findings that block; one advisory finding. FINDING — src/kiro_crew/decisions/points/model_route.py:367 — [OPUS-REVIEWED] 0893809 Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
UX Review (Fable 5) — 🟡 CONCERNSUX-level review of UX-Verdict: CONCERNS "Auto (Jev)" sits beside "auto" with nothing telling them apart, and the shipped default receipt "(unpinned)" means nothing to a first-time reader. Watch
Suggestions
[UX-REVIEWED] 0893809 |
GPT 5.6 Review — ✅ no blocking findingsGPT 5.6 completed its review of This comment is updated in place on each push. Review detailsFINDING -- src/kiro_crew/decisions/points/model_route.py:244 -- False positive or not applicable? A repository writer can comment: |
|
[operator: iamwhatever] Re-captured after the owner changed the Captured by Both receipts under one reply, light and dark: A simple turn routed to haiku, light: Expanded, dark and light: The picker with And withheld — the org answer is false, so the entry is not offered: |
54a0dcb to
cd77ce8
Compare
cd77ce8 to
69bce07
Compare
|
[operator: iamwhatever] fixed — The finding. The Legitimate. Yes, and the reasoning is the feature's own. Fixed, not exempted. One thing the fix adds, because the empty default made it reachable: I did not widen the gate. Note for the record that the spec's own Tests. Docs. Spec § 2 and § 9 and Owner note. This reverses the default values I was asked for one round ago (
|
|
[operator: iamwhatever] Re-captured for the ruled shape: every tier ships unpinned, so the row that most readers see first is
The shipped state — the tier was answered, that tier is unpinned, the turn kept its session's model: A reply decided by both questions, with the model tier pinned: Expanded, dark and light: The picker with And withheld — the org answer is false, so the entry is not offered: |
69bce07 to
d489286
Compare
|
[operator: iamwhatever] fixed — The shape. An unpinned tier is answered, not skipped. It records Ids removed.
Tests. Evidence re-captured for the shipped state (the unpinned row is the one most readers see first) and re-attached above; the description points at those URLs. Local: 381 python tests in the decisions/config/slot scope, 88 website tests, |
|
[operator: iamwhatever]
|
|
[operator: iamwhatever]
|
|
[operator: iamwhatever]
|
|
[operator: iamwhatever]
|
|
[operator: iamwhatever]
|
|
[operator: iamwhatever]
|
|
[operator: iamwhatever]
|
|
[operator: iamwhatever]
|
|
[operator: iamwhatever]
|
|
[operator: iamwhatever]
|
|
[operator: iamwhatever]
|
|
/ai-review override gpt b86057e: CI fully green; the substantive findings (owner-only arming, in-flight manual pick wins, no auto:jev on a new slot, owner-only routing, applied=false record integrity) are fixed; the two remaining items (context window of an owner-pinned model, Auto entry hidden on a failed consent read while the manual picker stays) are owner-only edge cases tracked in #12289. |
Human judgment recorded@iamwhatever marked the gpt AI finding as false positive, not applicable, or explicitly accepted for
This decision applies only to this commit. A new push requires a new judgment. |
|
/ai-review override gpt 8d5a159: rebase of the already-overridden head b86057e onto main; CI fully green; substantive findings fixed (owner-only arming and routing, in-flight manual pick wins, no auto:jev on a new slot, applied=false record integrity); remaining owner-only edge cases tracked in #12289. |
Human judgment recorded@iamwhatever marked the gpt AI finding as false positive, not applicable, or explicitly accepted for
This decision applies only to this commit. A new push requires a new judgment. |
|
/ai-review override gpt e0478f4: rebase onto main after #12253; CI fully green; the one new finding (manual pick via a session alias during the Jev await) is the narrower alias-path variant of the already-fixed in-flight-pick race, owner-only, recorded in #12289; all earlier substantive findings remain fixed. |
Human judgment recorded@iamwhatever marked the gpt AI finding as false positive, not applicable, or explicitly accepted for
This decision applies only to this commit. A new push requires a new judgment. |
bolichen97
left a comment
There was a problem hiding this comment.
Approving. Same consent shape as #12338 and it holds for the same reasons: off by default, the owner must pick Auto (Jev) in the model picker per session, and a manual model pick clears the flag so a concrete pin is never overridden by a tier. Cron deliveries, sub-agent turns, crew-relayed turns, app injections and autonudge wakes are excluded at the call site, each already resolving its model through its own tier — and the stated reason is the right one: nobody is watching the price of those answers at the moment they run.
Two design calls worth naming:
- Every tier ships as
""rather than a model id.model-selection.mdforbids hardcoding one, and the concrete reason is good: an id an account is not entitled to fails on the first prompt rather than when it was set. So the operator copies the example and fills in what their own picker offers. - An unpinned tier is answered, recorded and shown —
complex → (unpinned)— rather than treated as a refusal. That is what makes the feature observable before anyone commits to it: an owner can watch which tier their turns land in for a while, then pin only the levels worth moving. Collapsing"auto"and a non-string to""so "inherit" has one spelling is the right normalization.
Refusal leaves the session on the model it was already using, including when the mapped id is one the account cannot run, and the log distinguishes the two so nothing is dropped silently.
The gpt override is accepted: it carries forward an already-overridden head rebased onto main, the substantive findings (owner-only arming and routing, in-flight manual pick wins, no auto:jev on a new slot, applied=false record integrity) are fixed, and the one new item is the narrower alias-path variant of the already-fixed in-flight-pick race, owner-only and recorded in #12289.
|
Approved above, and it was next in the merge queue — but #12338 landed one minute earlier and took the same seam, so this went dirty before it could merge. Flagging so the approval is not read as ready against the current base.
The two that decide the rest:
Everything else ( Two notes before the push:
Say the word and I will do the rebase and run the decisions suites; I would rather you re-issue the override yourself. |
GPT 5.6 Review — ✅ no blocking findingsGPT 5.6 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
A chat slot runs every turn on one model: the id the owner pinned, or the backend's default. That is a single setting for a conversation whose turns are not one difficulty -- "fix this typo" and "design the migration" are the same slot and the same price. Adds a second decision point on the shipped Jev seam. `model.route` asks the oracle to put ONE turn in one of three tiers (simple / medium / complex) and runs that turn on the model `decisions.model_route` maps the tier to. A refusal -- no consent, unsampled, an answer outside the three tiers, an id this account cannot run, a timeout, a provider failure -- leaves the session on the model it was already on, which is what every unconsented install does. Never behind the owner's back. The chat model picker carries an `Auto (Jev)` entry, drawn only while the fleet's `decisions_enabled` AND the owner's keystone consent are both an exact `true`, and selecting it records `slot.jev_route`. Any concrete pick clears the flag: picking a model by hand is the owner answering the very question this point asks. Only a normal dashboard chat turn is routed -- cron deliveries, sub-agent turns, crew-relayed turns, app injections and autonudge wakes each already resolve their model through their own tier. Reuses the shipped seam rather than adding a second one: the same gate, the same consent keystone, the same bucket, the same JSONL log, the same strip, the same feedback route and side. Three things were widened to carry a second point: `DECISION_POINT_NAMES`, the outcome registry (keyed per point, so two decisions on one turn both reach the reply), and the strip record (a list, dispatched on each record's own `point`; a bare object still reads as one record). The sentinel `auto:jev` is resolved at the top of the slot-model handler and nowhere else, so every reader below it -- the guard, `session/set_model`, the reset, the session allocation, the composer chip -- takes the plain `auto` the session actually runs on. It is deliberately not stored in `slot.model`: that field is a provider model id, and a value no provider advertises would need a filter at each of those readers. Arming answers to the OWNER, not merely to an authenticated caller: the slot-model route's cross-app guard admits an allow-listed identity whose `app` claim is empty, so the `auto:jev` branch alone is gated on `is_owner_dashboard_request`, and a fork inherits the flag only for the owner. A plain pick on the same route is untouched. Provenance, not the actor, decides whose turn is routed. A dispatch that names no actor falls back to `user` by design, so the rewind, regenerate and OpenAI-compatible paths arrive as `user` while carrying an app's provenance -- which would have billed the owner for a routed turn nobody typed. The guard asks for `_directive_user_origin`, the flag derived from the auth middleware's app claim, and keeps the actor check beside it for the wakes that do declare themselves. Two ordering rules make the apply path honest. The answer's premise is re-read inside the two locks -- the flag, and the model the baseline named -- so a pick made by hand during the oracle's network round trip wins and the answer is dropped rather than applied over it. And the picker entry requires a live slot: without one a pick is held as the pending model and forwarded into slot creation, which would store the sentinel as a new slot's model with routing off. A returning `set_model` is not proof of a switch. A backend that judges the model VALUE can exhaust its candidate ladder and stay on its own default without raising, so the model the session reports is read after the switch and recorded: `applied` is false when the turn neither reached the chosen id nor moved off its baseline at all, `model_used` names what it ran on, and the reply's routing line says `not applied -- ran on <model>`. The row is append-only and is what an owner reads to decide what to pin, so it states what happened rather than what was asked for. Cost note: routing to a dearer model spends more. That is why the point is reachable only through the picker entry -- the owner opted in by selecting it, and clearing it is one pick. Tests: the apply path is driven through the real runner hook against a recording client, with two mutation checks beside it (break the map, change the answer) so a hook that ignored either fails rather than passing by coincidence. Plus the tier question and its bounds, every refusal, the error rows, the reader and the strip row, and the picker's two fail-closed gates.
buluoray
left a comment
There was a problem hiding this comment.
Verified at head 0893809c. Approving.
The question that decides this PR is what leaves the machine, and it is verbatim user text: model_route.build_state() sends {"message": (text or "")[:2000]} plus optional history to https://api.typesafe.ai/v1/systemone via gate.py:610. I traced the chain in front of it rather than assuming it:
- Consent lives in the sealed
decisions_consent.json, not the agent-writableconfig.json, and unreadable means off. Then the governance ceiling, the point allowlist, and the sample bucket, thenscrub_reasonover the rendered state, which refuses the whole request on any credential or exfil hit and on a scan failure. - History is restricted structurally rather than by convention:
build_historywalks onlyHISTORY_ROLES = frozenset({"user", "assistant"}), so tool output and code are never projected. The default budget is 0 and the ceiling ismin(config, consented_ceiling). - The call site fires only on
slot.jev_route,_directive_user_origin,_crew_log_actor == "user", and not synthetic/slash/recovery — and it sends_jev_route_textcaptured before any runner prepend, so no cancelled-turn preamble or drained app context rides along. - Arming "Auto (Jev)" is owner-only: a non-owner pick gets
_owner_denial_responseplus a SEL row.
I gave the -103 in skills_select.py its own pass, since a large deletion in a decision surface is where a guard quietly disappears. It is a pure extract-to-shared-module: build_history, _prior_turns, _history_budget, MAX_HISTORY_MESSAGES and HISTORY_ROLES moved into points/__init__.py, and the load-bearing part is byte-identical in its new home — HISTORY_ROLES itself and the if role not in HISTORY_ROLES: continue skip. Both points now share the one guard. Nothing was removed.
Both earlier blocking findings are genuinely fixed here rather than force-pushed away. jev_route is memory-only (state.py __slots__, init False) and all three restore paths explicitly refuse to read it back from the agent-editable transcript, so a prompt-injected agent cannot self-arm paid routing; fork inheritance is owner-gated. And _route_model_for_turn re-reads both premise halves inside the locks, so a manual pick during the round trip drops the routed answer. The two dismissed human reviews were dismissed by the rebase — one was an approval, the other is empty.
No hardcoded model id: all tiers default to "", "auto" normalises to "", and the only frontend model literal is an illustrative example inside a comment. The frontend cannot invent a tier — jevRoute.ts says in its own docstring that neither read is an enforcement point, and decisionStripContract.test.ts positively forbids api_key, secret, prompt, message and content from the strip payload.
One nuance worth stating rather than blocking on: adding model.route to DECISION_POINT_NAMES means a consent given when only skills.select shipped also authorises this point to send the 2000-char excerpt. The payload category is the same one that consent covered, and this point additionally needs the per-slot owner arm that did not exist before, so no turn is sent without a fresh explicit owner action. Defensible — but the base keystone is reused rather than re-prompted, and a reviewer should know that.

















Problem / Motivation
A chat session runs every turn on one model. "Rename this variable" and "redesign the scheduler" are the same session, so they get the same model and the same price. A user who wants the cheap one for small asks has to change the model by hand, remember to change it back, and pay for the times they forget.
Why it matters
The two directions both cost something. Staying on a strong model makes every small ask expensive. Staying on a cheap one makes the hard asks worse. Nobody wants to drive the picker by hand between messages, so most people pick one model and live with the wrong answer half the time.
What changed (motivation → approach → change)
You can now let Jev pick the model for each message. Open the model picker under the chat box and choose Auto (Jev). From then on Jev judges each message as
simple,mediumorcomplex, and that message runs on the model you mapped that level to. Small asks can go cheap; hard asks can go strong. A message a scheduled job, a sub-agent or an app sends is never routed.Auto (Jev)in the model pickerdecisions.model_routeinconfig.jsonThis rides the Jev seam that already ships rather than building a second one: the same gate, consent file, sampling, log, receipt and thumbs route. Three things were widened so a second question fits. The point list gained
model.route. The receipt registry is keyed per question, so a reply decided twice carries both instead of the second erasing the first. The receipt is a list, and which row is drawn comes from each record's ownpoint.Auto (Jev)is not a model. Picking it sendsauto:jev, which the gateway turns into plainautoat one place and remembers as a flag, so everything downstream sees a normalauto.When Jev is slow, unavailable, or answers outside the three levels, the message runs on the model the session was already using — what happens today for everyone. The log says which, so "Jev answered and nothing happened" is visible rather than silent.
Not yet on the receipt: a level pinned to a model this account cannot run keeps the session's model and writes only a log row — the failure rendering has no producer. Tracked in #12289.
Cost: routing to a stronger model spends more. So only the OWNER can turn it on: a non-owner is refused, a fork inherits it only for the owner, and the entry needs a live chat. A pick made by hand always wins, even mid-answer.
No model id ships as a default. Every level of
model_routeis""— inherit — until an owner pins one, and"auto"means the same.model-selection.mdrequires it: a hardcoded id fails on the first prompt for anyone not entitled to it. Values go throughnormalize_agent_model, the normalizeragent.role_modelsuses.The choice is never read back from disk. It is an owner pick that spends money and transcript metadata is agent-editable, so persisting it would let a prompt-injected agent write
"jev_route": true, restart, and be granted routing nobody selected. The flag is in-memory only, so a restart leaves the chat on its usual model and the owner re-picks. The entry is withheld for a remote-crew session too, whose turns run on the peer and never reach the hook.An unpinned level is still answered. Not a refusal: it is asked, logged and shown as
simple → (unpinned), and the turn keeps its model — so a new install can read where its messages land before pinning anything. An unpinned level and a pin the account cannot run both keep the model, andis_unpinnedtells them apart so only the second records an error.Where each piece lives:
src/kiro_crew/decisionspoints/model_route.py;gate.pylists the question and reads the map;outcomes.pykeys receipts per question; the prior-turn read moves intopoints/__init__.pyso both questions share one budget, andpoints/skills_select.pyreads it from theresrc/kiro_crew/dashboardchat_handlers.pyresolves the picker id and commits the flag on each success path only;chat_runner.pyasks the question and applies the model;state.pyholds the flag,slot_projection.pyreports it,chat_fork.pyinherits it, andchat_persistence.py/channel_slots.pydeliberately neither save nor reload itsrc/kiro_crew/configsections.pyaddsdecisions.model_routewith all three levels present and"", each normalized throughnormalize_agent_model;config-baseline.jsonregeneratedsrc/kiro_crew/context.pywebsite/src/pages,website/src/components,website/src/lib,website/src/typeslib/jevRoute.tsdecides when the entry is offered;ChatPage.tsxandcomponents/ChatPane.tsxdraw it;components/ModelDropdownList.tsxshows its label;chat/decisionRecord.ts,chat/DecisionStrip.tsxandchat/AssistantMessage.tsxread and draw the receipts;settings/decisionsPreview.tsandtypes/index.tsname the question and the flagwebsite/src/i18nwebsite/scriptscapture-decision-model-route.mjs, the screenshot capture behind the shots below: it drives the real built dashboard with the network stubbed and asserts what it photographs. No shipped surface of its owndocs,src/kiro_crew/docsdocs/system-specs/modules/decisions.md§ 9 is the contract;docs/feature-map/README.mdandsrc/kiro_crew/docs/decisions.mdsay how to turn it onBackwards compatibility
Compatible: no turn changes model until an owner picks
Auto (Jev)— which needs the Decisions consent that is itself off by default — AND pins a level indecisions.model_route, where every level ships unpinned.decisions.model_routeis a new optional key whose absence pins nothing. The receipt on a reply is now a list; the dashboard reader accepts the single object earlier replies carry, so old transcripts render unchanged. No field became required, no validator got stricter, and no key was renamed or removed.The receipt reports what happened, not what was asked for. A switch can be accepted and still not take — some backends judge the model id itself and keep their own default — so the reply says
not applied — ran on <model>, and the recorded row says the same. That row is never rewritten and an owner pins from it.One limit worth stating plainly: picking a model by hand stops the routing, and it does not move the session back to the model it was on before routing started. A routed session stores
auto, so choosing plainAutoto switch routing off takes the same-value path and changes no model. The revert — captured when routing is armed, restored when it is switched off or when a tier is unpinned — is tracked in #12289.Tests
test/test_decisions_model_route_apply.pydrives the real turn runner against a recording client and asserts the model id the provider is handed. Two mutation checks sit beside it — one changes the level-to-model map, one changes Jev's answer — so a hook that switched to a fixed model, or switched on nothing, fails instead of passing by luck. Plus every refusal (not picked, no consent, unsampled, an id the account cannot run, a failing switch, no switch seam) asserted as "the provider was never asked to switch", and the five turn kinds that are never routed.test/test_decisions_model_route.pycovers the question and its bounds, reading the answer, the map, resolving a model against the advertised list, prior turns under the shared budget, and the error rows. It pins that every level ships unpinned, that"auto"and a non-string both collapse to inherit, that the shipped map still answers and applies nothing, and that an unpinned level is told apart from a pin the account cannot run. Two tests guard the comparison: two ids one character apart are never folded together, and the registry's dotted/dashed pair stays separate. Tests use placeholder ids (model-a); the registry pair is the one exception, because a placeholder has no registry entry to test with.website/src/test/DecisionStripModel.test.tsxcovers the reader and the row: a record without a level draws nothing, an EMPTY model is accepted and rendered as(unpinned), a record is dispatched on its ownpointand never read as the other kind, and a reply decided twice draws two rows.website/src/test/jevRoute.test.tsxcovers the picker's two gates as fail-closed, the row's position, that it shows a label and not the wire id, and that a routed session highlights that row rather thanauto.website/src/test/decisionStripContract.test.tsnow also parses the model record out of the spec, so the two halves cannot drift apart silently.Manual verification
website/scripts/capture-decision-model-route.mjsruns the real built dashboard with the network stubbed and asserts as well as photographs: the row is the picker's first entry, a routed session reads as selected on it, the row is absent when the org answer is false, and a past routed turn still shows its receipt then. All assertions pass. A real Jev call needs an API key and consent to send test data, so the routing decision itself is covered by the unit and runner tests above.Screenshots / video
The shipped state — the level was answered, that level is unpinned, the turn kept its session's model:
A reply decided by both questions, with the level pinned:
The model picker with
Auto (Jev)besideauto, selected:Opened, the row shows the level, both models and what the question carried:
A switch that was accepted and did not take — the row names the model the turn ran on, not the one it asked for:
Opened, "Model used" agrees with that line rather than crediting the model that was never used:
The row's failure rendering, drawn from a hand-written record rather than a live one — nothing publishes this state today:
record_errorwrites the log row and deliberately does not publish, so this is what the copy will look like once it does. The missing producer is tracked in #12289; the frame is here because the copy is reviewable now:Every variant
And the withheld case — the org answer is false, so the entry is not offered at all:
Related Issues
no linked issue: this is the second decision point on the Jev seam, dispatched alongside #12245 (strip polish) and #12253 (tool risk); none of the three has a tracking issue.
Checklist