Skip to content

fix(settings): save provider/model ids in every model pin its picker offers - #18399

Merged
chenmingwei23 merged 1 commit into
mainfrom
r23-16768-qualified-pins
Oct 11, 2026
Merged

chenmingwei23 merged 1 commit into
mainfrom
r23-16768-qualified-pins

Conversation

@iamwhatever

@iamwhatever iamwhatever commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A model you pick from a Settings dropdown your account advertises can be saved — in every model pin whose runtime can use it, not just the Decisions tiers.

#16417 widened the Decisions tier pickers' save grammar to accept the provider-qualified provider/model ids OpenCode advertises. It left five sibling pins — agent.model (the Settings → Chat default), agent.role_models.background, agent.role_models.subagent, agent.fallback_model and agent.refusal_fallback_model — on the narrow ^[A-Za-z0-9._\-\[\]]*$. Those pins' pickers list the same advertised names, so on OpenCode selecting one and saving answers "Could not save this setting" for a model the picker itself offered.

Why it matters

A user picks a model from the dropdown their own account advertises, hits save, and the PATCH rejects it. The picker and the save gate disagree about what a valid id is — a dead end with no workaround short of hand-editing config.

Not a goal

  • No change to decisions.nudge_wake.llm_model: it keeps the bare-id grammar, because the decision gate scrubs any model decisions.types.is_model_id refuses (no /), so a saved provider/model wake-judge pin would never reach the judge. Widening the gate's MODEL_ID_RE is out of scope.
  • No change to the entitlement check — _validate_role_model still rejects a well-formed id the active provider or the account cannot honor.
  • No change to load-time config handling.

What changed (motivation → approach → change)

Symptom: five Settings pins reject an id their own picker offered. Root cause: #16417 widened only the tier pickers' pattern; the siblings kept the pre-OpenCode narrow pattern, so the two drifted apart. Change: give those pins and the tiers one shared grammar constant (agent.model takes the grammar only; it has no entitlement validate_fn), _MODEL_PIN_PATTERN in dashboard/handlers/core.py — a bare id OR the slash-joined provider/model form, still refusing empty segments (a//b, /a, a/) and shell metacharacters. The tier picker that #16417 added now references the same constant.

Backwards compatibility

Mostly widening: provider/model ids now pass on the five pins, and every bare id that starts with a letter or digit passes as before. One narrowing: the shared grammar (already the tiers' grammar since #16417) requires the first character to be a letter or digit, so an id starting with ., _, - or [ — which the old narrow pattern allowed — is now refused on these five pins. No advertised model id has that shape; the tiers have refused it since #16417.

Removes nothing: no path is hidden, no record pruned, no key or function removed.

Tests

New test/test_qualified_model_pins.py, for each of the four validated pins:

  • a provider-qualified id (example-provider/example-model) is stored exactly as sent;
  • malformed segments (provider//model, /model, model/) and a shell-metacharacter value are refused with HTTP 400 and nothing is stored;
  • the pin shares the tier picker's pattern and validate_fn, so the two cannot silently drift apart again.

For agent.model: a provider/model id saves, the same malformed values get 400, and it shares the tier pattern (fails on base).

And for decisions.nudge_wake.llm_model: a provider/model id is refused with 400 (it would fail is_model_id, so the judge would be scrubbed), and a bare id still saves.

All 32 cases pass locally, plus test_config_patch.py, test_config_live.py, test_config_overlay.py, test_model_scope.py, the config hot-reload suites and test_decisions_config_api.py (471 total).

Manual verification

N/A — the PATCH handler is exercised end-to-end by the new unit tests (real handler against a temp config.json).

Related Issues

Fixes #18396

Replaces #16768 by @leonlaiyc.

Pattern harvest

Not generalizable: a one-off grammar drift between a picker's save gate and its sibling pins, fixed by collapsing them onto one shared constant. No wider pattern worth a lint/semgrep rule — the shared-constant structure is itself the guard against recurrence (the test asserts the pins reference the tier's pattern).

@iamwhatever
iamwhatever requested review from a team and KiroCrewCaptain as code owners October 9, 2026 08:21
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 9, 2026
@github-actions

github-actions Bot commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision b2cf1a746ecf7e886e2171addcc31fa1fd4dadbe touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Advisory — does not block merge.

@github-actions

github-actions Bot commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of b2cf1a746ecf7e886e2171addcc31fa1fd4dadbe — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Nothing to check.

[DESIGN-REVIEWED] b2cf1a7

Description read

[DESCRIPTION-READ] b856ed2cbd9b799c9c23b821d850108bb189f79ddf5ff75a2067e24f604682f1

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of b2cf1a746ecf7e886e2171addcc31fa1fd4dadbe — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

On OpenCode, the Decisions wake-judge picker still lists provider/model names that its save refuses. That dead end is declared out of scope; confirm the deferral is intended.

Not justified as shipped

  • Item 4 — rides along: the narrowing comes from sharing one grammar, and "No advertised model id has that shape" is asserted without evidence. It is reversible, and validation.MODEL_ID_RE already requires a leading letter or digit.

What this change ships

Inventory (6 items) — 5 justified

Intent: let a model picked from a Settings dropdown be saved in every pin whose runtime can use it. This is a FIX (#18396).

  1. Settings → Chat default model now saves provider/model ids — justified
  2. Background and subagent role-model pins now save provider/model ids — justified
  3. Throttle-fallback and refusal-fallback pins now save provider/model ids — justified
  4. Ids starting with ., _, - or [ are now refused on these five pins — rides along, a consequence of the shared grammar
  5. The tier pickers' grammar is now one shared constant (same behaviour) — justified
  6. model-selection.md and decisions.md describe the shared grammar and the wake-judge exception — justified

[FIRST-PRINCIPLES-REVIEWED] b2cf1a7

Description read

[DESCRIPTION-READ] 6454d3f8bb05f15195ab506efa05a52987f56e0640312a0e2346731f446a0ae7

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of b2cf1a746ecf7e886e2171addcc31fa1fd4dadbe and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] b2cf1a7

False positive or not applicable? A repository writer can comment:
/ai-review override gpt b2cf1a746ecf7e886e2171addcc31fa1fd4dadbe: <one-sentence reason>

@github-actions

github-actions Bot commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed b2cf1a746ecf7e886e2171addcc31fa1fd4dadbe — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] b2cf1a7

Verdict parsed from the review's SHA-scoped output markers for commit b2cf1a746ecf7e886e2171addcc31fa1fd4dadbe.

False positive or not applicable? A repository writer can comment:
/ai-review override fable b2cf1a746ecf7e886e2171addcc31fa1fd4dadbe: <one-sentence reason>

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 9, 2026
@leonlaiyc

Copy link
Copy Markdown
Contributor

Confirmed the Opus advisory on head 7c36e3c0708a7ff4360139263d93bd7b77a45eef with the exact-head Settings pattern and scrub_reason function:

model = "example-provider/example-model"
Settings _MODEL_PIN_PATTERN.fullmatch(model): True
is_model_id(model): False
scrub_reason({}, [], model=model): "scrubbed:provider-model"

decisions.nudge_wake.llm_model saves successfully, but lane_model(..., lane=LANE_LLM) passes the saved id into the decision gate's scrub, which returns before invoking the judge. The user sees a saved pin that the wake judge cannot use.

The two remediation options described in the review are to retain the bare-id grammar for this pin (and align its spec/tests), or to accept the same bounded slash-segment grammar in the runtime model validator. Offering this reproduction as evidence for whichever contract you choose; I have not changed this branch.

@iamwhatever

Copy link
Copy Markdown
Contributor Author

Self-review (crew-pr-reviewer, read-only): CHANGES NEEDED

This PR moves every Settings model pin onto one _MODEL_PIN_PATTERN grammar, so provider/model ids that OpenCode advertises can be saved. For the role-model and fallback pins that fixes the "could not save" error at its cause. For one pin it creates a silent regression. The Opus lane's advisory on head 7c36e3c is real, and leonlaiyc reproduced it on the exact head. decisions.nudge_wake.llm_model (src/kiro_crew/dashboard/handlers/core.py:3051) now saves example-provider/example-model with a 200. But gate.lane_model passes that id to scrub_reason, and there is_model_id (MODEL_ID_RE in decisions/types.py:23, no / allowed) returns scrubbed:provider-model. So Settings shows a saved pin that the wake judge never runs on. That advisory has no author disposition yet. Suggested fix, pick one: keep decisions.nudge_wake.llm_model on the bare-id pattern and drop it from the spec text in docs/system-specs/modules/decisions.md / common/model-selection.md and from test/test_qualified_model_pins.py; or widen MODEL_ID_RE to the same bounded slash-segment form, with a test that a qualified wake-judge pin reaches the judge. I checked the diff, the new tests, all lane verdicts and the reproduction comment. CI is green. — crew-pr-reviewer

@iamwhatever
iamwhatever marked this pull request as draft October 9, 2026 16:34
@iamwhatever

Copy link
Copy Markdown
Contributor Author

Moved to draft by crew-pr-reviewer: the self-review above is CHANGES NEEDED. GitHub refuses a self-request-changes and the readiness check ignores human reviews, so draft keeps this PR from merging (it also turns auto-merge off). Once the points above are fixed, mark it ready with gh pr ready 18399 and it will be re-reviewed on the new head. — crew-pr-reviewer

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: passed Eligible automated validation passed for the current revision labels Oct 9, 2026
@iamwhatever
iamwhatever force-pushed the r23-16768-qualified-pins branch from 7c36e3c to 313a3f8 Compare October 10, 2026 20:18
@iamwhatever

Copy link
Copy Markdown
Contributor Author

Addressed the CHANGES NEEDED self-review (and the Opus advisory it confirmed) on new head 313a3f8264ef2992fb2e1ff9bc989631af18fc24 (rebased once onto current main, still one commit, authorship kept):

  1. decisions.nudge_wake.llm_model silent regression — took the first suggested option. The wake-judge pin is back on the bare-id grammar ^[A-Za-z0-9._\-\[\]]*$ (src/kiro_crew/dashboard/handlers/core.py), with a comment explaining why it does not share _MODEL_PIN_PATTERN: the decision gate scrubs any model decisions.types.is_model_id refuses. MODEL_ID_RE is unchanged.
  2. Spec text — docs/system-specs/common/model-selection.md and docs/system-specs/modules/decisions.md no longer list the wake-judge pin under the shared grammar; both now state it keeps the bare-id grammar and why.
  3. Tests — decisions.nudge_wake.llm_model is removed from _MODEL_PINS in test/test_qualified_model_pins.py. Two new tests: a provider/model wake-judge pin is refused with 400 and nothing is stored (this one fails on the previous head with 200), and a bare id still saves and passes is_model_id.

Also corrected the PR body: it now names four pins, lists the wake-judge pin under "Not a goal", and fixes the Design Review note — the shared grammar refuses an id starting with ., _, - or [ on these four pins, so "Nothing main accepts today is rejected" was not accurate.

Local: test_qualified_model_pins.py 26 passed; with test_decisions_config_api.py and test_decisions_judge_llm.py, 314 passed. black / isort / flake8 / mypy clean on the touched files.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 10, 2026
@iamwhatever
iamwhatever force-pushed the r23-16768-qualified-pins branch from 313a3f8 to 008a1b7 Compare October 10, 2026 22:02
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 10, 2026
@iamwhatever
iamwhatever marked this pull request as ready for review October 10, 2026 22:47
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 10, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor Author

Self-review (crew-pr-reviewer, read-only): CHANGES NEEDED

This PR moves the background, subagent, throttle-fallback and refusal-fallback pins onto the shared _MODEL_PIN_PATTERN, so a provider/model id their picker offers can be saved. For those four pins that fixes the cause, and the new tests would fail on base. iamwhatever's earlier request is addressed: decisions.nudge_wake.llm_model is back on the bare-id grammar, the specs say why, and a test proves a qualified wake-judge pin gets a 400. On head 008a1b7 two things are still open. First, the Opus advisory is real and has no reply: the comment at src/kiro_crew/dashboard/handlers/core.py:3017-3018 still says "a tier also accepts provider/model ids, while a role_models pin stays one segment", which this diff makes false. Replace it with "The grammar is _MODEL_PIN_PATTERN, shared with the role-model and fallback pins." Second, the First Principles CONCERNS is right and has no reply. The body's Not-a-goal says agent.model "has no provider-qualified picker", but website/src/pages/settings/ChatPanel.tsx:1028 builds its options from the same useAvailableModelsQuery() list, so on OpenCode that picker still dead-ends. Either widen agent.model too, or give the real reason it must stay narrow (it looks like acp/runtime.py:1116 MODEL_ID_RE refuses / for the chat runtime) in the body and in a disposition. Reviewer note: I checked that re.fullmatch applies the pattern (core.py:3216), so the unanchored start is safe. resolve_model stores role pins verbatim into agent specs. Please confirm that no consumer of these four pins passes them to AcpRuntime(model=...), which would refuse /. — crew-pr-reviewer

@iamwhatever
iamwhatever marked this pull request as draft October 11, 2026 00:23
@iamwhatever

Copy link
Copy Markdown
Contributor Author

Converted to draft by crew-pr-reviewer: the self-review above is CHANGES NEEDED. Mark ready again (gh pr ready 18399) once the points are addressed.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: passed Eligible automated validation passed for the current revision labels Oct 11, 2026
@iamwhatever
iamwhatever force-pushed the r23-16768-qualified-pins branch from 008a1b7 to 89c3ab6 Compare October 11, 2026 01:07
@iamwhatever

Copy link
Copy Markdown
Contributor Author

Reply to the 2026-10-11 00:23Z self-review. Fixed on head 89c3ab6 (rebased onto current main, still one commit, cixuuz + Co-authored-by kept).

  1. Stale comment at core.py (tiers block): replaced with "The grammar is _MODEL_PIN_PATTERN, shared with the role-model and fallback pins." This also answers the Opus advisory.

  2. agent.model / First Principles CONCERNS: you were right. ChatPanel.tsx:1028 builds its options from the same useAvailableModelsQuery() list, so the "no provider-qualified picker" claim was wrong. agent.model now uses _MODEL_PIN_PATTERN too (grammar only; it has no entitlement validate_fn, unchanged). New tests: a provider/model id saves (fails on base with "invalid value for agent.model"), malformed values (provider//model, /model, model/, shell chars) get 400, and the pin shares the tier pattern. The comment above the key, the spec (model-selection.md) and the PR body are corrected; the wrong Not-a-goal line is gone.

  3. AcpRuntime(model=...): no production code passes a pin there. Every construction site (providers/acp.py:1259 and :1409, session_background.py:770, code_review_sage/.../review_pool.py:506) omits model=, so the MODEL_ID_RE check at acp/runtime.py:1124 never sees any of these pins. It is only a defensive check on an unused argument. No code was added for it.

Local: 32/32 in test_qualified_model_pins.py, 471 across the config PATCH / live / overlay / model-scope / hot-reload / decisions-config suites; black, isort, flake8 clean.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 11, 2026
…offers

The role-model, throttle-fallback and refusal-fallback model pickers list
the same advertised model names as the Decisions tier pickers, and OpenCode
advertises those names as `provider/model`. #16417 widened only the tier
pickers' grammar; the four sibling pins still used the narrow
`^[A-Za-z0-9._\-\[\]]*$` and answered "Could not save this setting" for a
model their own picker offered.

Give those pins and the tiers one grammar, `_MODEL_PIN_PATTERN` in
`dashboard/handlers/core.py`: a bare id OR the slash-joined `provider/model`
form, still refusing empty segments and shell metacharacters.

`decisions.nudge_wake.llm_model` keeps the bare-id grammar: the decision
gate scrubs any model `decisions.types.is_model_id` refuses (no `/`), so a
saved `provider/model` wake-judge pin would never reach the judge.

Co-authored-by: Leon <276034861+leonlaiyc@users.noreply.github.com>
@iamwhatever
iamwhatever force-pushed the r23-16768-qualified-pins branch from 89c3ab6 to b2cf1a7 Compare October 11, 2026 03:09
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 11, 2026
@iamwhatever
iamwhatever marked this pull request as ready for review October 11, 2026 04:54
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 11, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor Author

Self-review (crew-pr-reviewer, read-only): RECOMMEND APPROVE

This PR moves agent.model, the two agent.role_models.* pins, agent.fallback_model and agent.refusal_fallback_model onto the tier pickers' grammar through one shared _MODEL_PIN_PATTERN constant. Those pins can now save the provider/model ids their own OpenCode pickers offer, and the save gate can no longer drift away from the picker, so the root cause is fixed. I read the full diff and confirmed that re.fullmatch applies the pattern (core.py:3217), so the missing ^ is safe. I also confirmed that no production AcpRuntime(...) construction passes model=, so the slash-refusing MODEL_ID_RE check at acp/runtime.py:1141 never sees these pins. I ran test/test_qualified_model_pins.py: 32 passed at b2cf1a7, and 10 failed with the base core.py, so the tests prove the fix. iamwhatever's earlier request is addressed: decisions.nudge_wake.llm_model keeps the bare-id grammar with a test, the stale tier comment is replaced, agent.model is widened, and the AcpRuntime(model=) question is answered. All 91 checks are green or skipped, there are no open reviews, and the GPT, Opus and Design lanes pass. Reviewer note: before merge, please answer the First Principles PASS punchline on b2cf1a7. The wake-judge picker still lists provider/model names that its save refuses, so the disposition should confirm that dead end is a deliberate deferral (ideally with a follow-up issue). It should also confirm that the leading-character narrowing (., _, -, [) on the five pins is acceptable. — crew-pr-reviewer

@iamwhatever
iamwhatever enabled auto-merge (squash) October 11, 2026 05:16
@chenmingwei23
chenmingwei23 merged commit 1821147 into main Oct 11, 2026
133 of 153 checks passed
@chenmingwei23
chenmingwei23 deleted the r23-16768-qualified-pins branch October 11, 2026 05:45
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(settings): five model pins reject the provider/model ids their own pickers offer

4 participants