Skip to content

fix(settings): save provider/model ids in every model pin its picker offers - #16768

Closed
leonlaiyc wants to merge 1 commit into
kirodotdev:mainfrom
leonlaiyc:fix/qualified-model-ids-in-model-pins
Closed

leonlaiyc wants to merge 1 commit into
kirodotdev:mainfrom
leonlaiyc:fix/qualified-model-ids-in-model-pins

Conversation

@leonlaiyc

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Every Settings model picker can save any model it offers, including the provider-qualified provider/model ids OpenCode advertises.

#16417 fixed this for the three Decisions (Jev) tier pickers. Five sibling pickers still hit it on main @ 63796306f: the two per-role model pins, the throttle-fallback model, the refusal-fallback model and the wake-judge model.

  • agent.role_models.background
  • agent.role_models.subagent
  • agent.fallback_model
  • agent.refusal_fallback_model
  • decisions.nudge_wake.llm_model

Every one of these pickers lists the same names as the tier pickers: useAvailableModelsQuery() → m.name, in website/src/pages/settings/ChatPanel.tsx (roleModelOptions, fallbackModelOptions) and DecisionsCard.tsx (judgeModelOptions). Their PATCH /api/config/kirocrew gate still uses ^[A-Za-z0-9._\-\[\]]*$, so on OpenCode picking an offered model answers "Could not save this setting" and the value reverts. The First Principles review of #16417 named these keys as the open siblings ("a follow-up issue tracks the same "/" rejection on agent.model, agent.role_models.* and decisions.nudge_wake.llm_model").

Why it matters

An OpenCode user cannot choose a cheaper model for background or sub-agent work, cannot set the model that a throttled or refused turn falls back to, and cannot pick the model that judges loop wakes. Each picker offers exactly those models and then refuses to keep them. Hand-editing config.json works, because the load path accepts the same ids. So the refusal is only the dashboard gate disagreeing with its own picker.

Not a goal

What changed (motivation → approach → change)

Symptom: a picker offers an id its own save refuses. Root cause: the grammar was copied per key, and #16417 widened only the tier copy, so the copies drifted. Every one of these keys already shares _validate_role_model, so only the pattern differed.

src/kiro_crew/dashboard/handlers/core.py:

  • New module constant _MODEL_PIN_PATTERN, byte-identical to the tier pattern fix(decisions): accept qualified models in Jev tier settings #16417 shipped (checked with == locally). It accepts a bare id, or slash-joined segments that each start with an alphanumeric, plus the empty value. It refuses a//b, /a, a/ and shell metacharacters. The PATCH applies it with re.fullmatch, and max_len: 64 is unchanged.
  • The five pins above and the tier loop now reference that constant. The comment that called the role pins "same grammar as agent.model", which is no longer true, is rewritten. The tier and nudge_wake comments already claimed parity with the role pins, and they are accurate again. That answers the other half of fix(decisions): accept qualified models in Jev tier settings #16417's clearing condition: its comments no longer describe a parity that does not exist.

Load-time compatibility was checked rather than assumed. coerce_role_models → normalize_agent_model keeps any non-auto string. coerce_fallback_model and coerce_refusal_fallback_model pass an unregistered id through model_registry.to_provider_id unchanged.

Specs, same commit:

  • docs/system-specs/common/model-selection.md gains a "save gate accepts every id its picker offers" rule beside the existing "Pickers MUST list from /api/models" rule.
  • docs/system-specs/modules/decisions.md's tier paragraph now says the wake judge and the role/fallback pins share the grammar.

Backwards compatibility

Breaking: for these five keys, a dashboard save of a value whose FIRST character is ., _, -, [ or ] is now refused (400). The old pattern admitted that shape and the tier pattern being reused does not. Every other value the old pattern accepted still matches the new pattern's first alternative, and _validate_role_model still judges entitlement for every non-empty, non-auto value. Writer sweep on origin/main @ 63796306f:

  • _EDITABLE_CONFIG's pattern is read in exactly one place, the PATCH handler (dashboard/handlers/core.py: lookup at :3089, re.fullmatch at :3142). Nothing re-validates a stored value with it.
  • Dashboard writers are the five Settings pickers, whose options are GET /api/models names plus auto and "". An advertised model id starting with punctuation would be refused here as it already is on the tier pickers.
  • Hand-edited config.json is read by the load-path coercers above, which apply no pattern, so an existing stored value of any shape keeps loading and working.

Removes nothing: no key, route or function is removed or renamed.

Tests

New test/test_qualified_model_pins.py runs the real api_kirocrew_config_patch handler against a temp config.json (the harness #16417's test_decisions_config_api.py uses), parametrized over the five pins:

  • test_a_provider_qualified_picker_model_can_be_saved[*]: example-provider/example-model → 200 and stored verbatim. Red on main for all five (400), green with the fix.
  • test_a_pin_refuses_a_value_outside_the_model_id_grammar[*]: a; rm -rf /, provider//model, /model and model/ → 400 and nothing stored. Green on main and with the fix (20 cases), so the widening opens nothing else.
  • test_every_model_pin_shares_the_tier_grammar[*]: each pin's pattern equals the tier's, and its validate_fn is the tier's. Red on main; it keeps the copies from drifting apart again.

Adjacent, local, with the fix: every test file that names these keys or the PATCH handler (53 files, -n 6): 5547 passed, 85 skipped, 2 xfailed, 4 failed. The 4 failures are test_chat_threads.py symlink tests that fail the same way on unmodified main (OSError: [WinError 1314], because creating a symlink needs elevation on the local Windows box); they never reach the config route. The tests that read the two edited specs (test_decisions_config_api.py, test_decisions_model_route.py, test_model_scope_wire_paths.py) pass: 133. black --check --target-version py310, flake8 and isort --check-only are clean on the changed files.

Manual verification

N/A — unit coverage sufficient: the defect is the PATCH gate's pattern, and the tests drive the real handler with the exact id shape OpenCode advertises. Not run end to end against a live OpenCode session. The picker-side claim (these pickers list useAvailableModelsQuery() names) is read off the source cited above. scripts/docs-lint.sh was not run (it is slow on this box). The two spec edits are prose inside existing sections, with no new link or file.

Related Issues

Follow-up to #16417: the sibling keys its First Principles review named. No separate issue was filed.

Pattern harvest

Rule candidate: review-prompt
Pattern: a validation grammar copied per key, one copy widened for a new id shape while sibling keys fed by the same picker keep the old copy; hoist the grammar to one named constant and pin every consumer to it.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

🤖 Generated with Claude Code

@leonlaiyc
leonlaiyc requested a review from a team as a code owner October 4, 2026 12:18
@leonlaiyc
leonlaiyc requested a review from cruisercohen October 4, 2026 12:18
@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: action required A blocking check or review needs attention labels Oct 4, 2026
@leonlaiyc
leonlaiyc force-pushed the fix/qualified-model-ids-in-model-pins branch from 732d739 to 68fb2dc Compare October 4, 2026 12:19
@github-actions

github-actions Bot commented Oct 4, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5, fork) — ✅ PASS

Design-level review of 4c55b7aa0e16cd5a01a1cd594c57adf910e0efdf via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Check that an OpenCode provider/model pin saved through these five keys reaches set_model and gets served at runtime; the tests only cover the PATCH gate.

[DESIGN-REVIEWED] 4c55b7a

Description read

[DESCRIPTION-READ] 1a933d4cd9ae878f8843994efa92c294a0db479561996489b57983f37df83c36

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 4, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5, fork) — ✅ PASS

Premise-level review of 4c55b7aa0e16cd5a01a1cd594c57adf910e0efdf via the fork AI-review pipeline — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

Before merge, check that no backend lists a model id starting with ., _, -, [ or ]. These five pins now refuse that shape.

Not justified as shipped

  • Item 2 — rides along: the stricter first character comes from reusing the tier pattern exactly. The description declares it ("a dashboard save of a value whose FIRST character is ., _, -, [ or ] is now refused").

What this change ships

Inventory (6 items) — 5 justified

Intent: let OpenCode users save the provider/model ids that the role, fallback and wake-judge pickers already offer. This is a FIX.

  1. Five Settings model pickers now save provider-qualified ids instead of failing with "Could not save" — justified
  2. Those five pins now refuse values starting with punctuation that they used to accept — rides along
  3. All six validated model pins now use one shared grammar constant, so the copies cannot drift apart — justified
  4. A test fails if any pin's pattern or validate_fn drifts from the tier's — justified
  5. The model-selection spec now says a pin's save gate accepts every id its picker offers — justified
  6. The decisions spec now says the wake judge and role/fallback pins share the tier grammar — justified

[FIRST-PRINCIPLES-REVIEWED] 4c55b7a

Description read

[DESCRIPTION-READ] dd0c31b552cde3ffb6e5bbd0ab85c908b71c2269c2d73e3ab4b5e7ce3e981604

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 4, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review (fork) — ✅ no blocking findings

Reviewed 4c55b7aa0e16cd5a01a1cd594c57adf910e0efdf via the fork AI-review pipeline; updated in place on each push.

Review details

No findings block this merge. One advisory: on the wake-judge pin, a provider/model id can now be saved but is never used.

FINDING — src/kiro_crew/dashboard/handlers/core.py:3024 — decisions.nudge_wake.llm_model now takes "pattern": _MODEL_PIN_PATTERN, so a provider/model id saves with a 200. At decision time, gate.lane_model(lane=LANE_LLM) returns that id unchanged and scrub_reason refuses it, because is_model_id (MODEL_ID_RE = r"\A[A-Za-z0-9][A-Za-z0-9._-]{0,63}\Z", no /) fails. Every LLM-lane wake decision is then written as scrubbed:provider-model and the judge never runs. Before this change the PATCH refused the id up front; now the failure is silent. → Fix: keep the slash-free pattern on decisions.nudge_wake.llm_model (revert that one hunk and its doc/test rows), or widen decisions.types.MODEL_ID_RE to the same slash-segmented grammar.

[OPUS-REVIEWED] 4c55b7a

@github-actions

github-actions Bot commented Oct 4, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review (fork) — ✅ no blocking findings

Reviewed 4c55b7aa0e16cd5a01a1cd594c57adf910e0efdf via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 4c55b7a

@leonlaiyc

Copy link
Copy Markdown
Contributor Author

On the two "before merge" checks from the current-head reviews (68fb2dc):

  • Design: can a slash-qualified pin be used, not just saved? Neither consumer applies a grammar of its own. The sub-agent pin is read by subagent._subagent_default_model through normalize_agent_model, which keeps any non-auto string, and is passed to the spawn as an explicit override. The throttle fallback goes through llm_helpers.configured_fallback_chain, which returns (fm, "auto"), into advance_fallback_candidate. That walk skips any id the live backend does not advertise and applies the first one whose set_model lands. The pickers list exactly the GET /api/models names, so on the backend that advertised a provider/model id, that id is applicable. On a backend that doesn't advertise it, the walk skips it and falls through to auto. That is what any unadvertised pin already does on main, so it is not new behaviour.
  • First Principles: does any backend advertise an id starting with ., _, -, [ or ]? This is already covered in the Backwards compatibility section. The tier pickers have refused that shape since fix(decisions): accept qualified models in Jev tier settings #16417, and these five pins now refuse the same shape.

Separately, CI: Backend Lint & Type Check (3.12) on this head is CANCELLED, not failed. mypy was still running at the job's 25-minute limit. The same cancellation hit #10952 and #14207 today, while main's own runs of that job take about 18 minutes. Nothing here changes type surfaces beyond one module constant. A re-run should clear it.

…offers

kirodotdev#16417 let the Decisions tier pickers save the provider-qualified
provider/model ids OpenCode advertises. The role-model, throttle-fallback,
refusal-fallback and wake-judge pickers list the same advertised names
(useAvailableModelsQuery) and validate through the same
_validate_role_model, but their PATCH gate kept the slash-free pattern, so
picking one of the offered models answered "Could not save this setting"
and reverted. Load-time config already accepts those ids.

Give every validated model pin one grammar, _MODEL_PIN_PATTERN (the tier
pattern, byte-identical), so a picker cannot offer an id a sibling pin
refuses. agent.model is unchanged: it has no entitlement check behind its
grammar.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@bolichen97
bolichen97 force-pushed the fix/qualified-model-ids-in-model-pins branch from 68fb2dc to 4c55b7a Compare October 6, 2026 09:30

Copy link
Copy Markdown
Contributor

Rebased onto latest main (5edf2feecb, was 268 commits behind).

  • Head: 68fb2dc792 → 4c55b7aa0e
  • Conflicts: none; git range-diff shows the single commit unchanged (=)
  • Verification: py_compile OK on changed .py files; pytest test/test_qualified_model_pins.py → 30 passed

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 6, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor

Thank you @leonlaiyc for this fix. Because the fork workflow needs maintainer approval to run its checks, I have opened #18399 on the main repo with your change (you are credited as co-author) so it can go through CI directly. Closing this in favour of #18399 — all the credit for the fix is yours. 🙏

@iamwhatever iamwhatever closed this Oct 9, 2026
@github-actions github-actions Bot removed the readiness: action required A blocking check or review needs attention label Oct 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants