Skip to content

feat: let an administrator allow or refuse each decision model - #36271

Merged
bolichen97 merged 1 commit into
mainfrom
feat/decision-model-allowlist
Nov 13, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
feat/decision-model-allowlist

Conversation

@bolichen97

@bolichen97 bolichen97 commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: An administrator can turn decision models off entirely or allow only specific ones, the gateway enforces it at every write and every use, and the dashboard says plainly which models the administrator does not allow.

Governance over decision models knows only two special cases today: the hosted host and the SageMaker host. An administrator cannot allow Laya and refuse everything else, nor ship an edition with the hosted provider refused while local models stay available. A policy that turns the row off makes the card vanish with no word of why, and a refused hosted option disappears from the picker instead of saying it is not allowed. The hosted provider's API key sits in a "Shared settings" block although only that provider uses it.

Why it matters

A managed fleet (the internal edition first) must be able to ship with one provider not allowed and have that hold in the gateway, not only in the UI, and every user must be able to read why a model is unavailable.

Not a goal

  • Renaming labels. The wording pass that calls the feature "decision model" everywhere (existing catalog values, backend strings, user docs, the transcript trace line) is a separate pull request opened after this one merges; this one changes only the governance wording it introduces.
  • Editing the internal edition's companion package. This PR provides the seam; the exact setting the edition ships is under "What changed".
  • Renaming or removing any stored identifier, config key, API field or ceiling key.
  • Putting the wake judge's small-model lane in the allow list: it runs on the session's own model provider and stays bound by the row and the points list, as models and endpoints already are.
  • Changing CONTRACT_VERSION, the evaluator, or the standalone default (no policy: every model allowed).

What changed (motivation → approach → change)

Approach. Generalize the existing capabilities.decisions ceiling instead of adding a parallel mechanism: one more inner list on the same row, asked wherever the row is already asked for the configured provider.

Policy keys and data shape.

  • capabilities.decisions.enabled stays the full off switch.
  • New inner list capabilities.decisions.scopes.providers (identifier matcher, a SCOPE_CATALOG data row). One entry per decision model, by the picker's own id: jev (hosted), plumb-4b, strands-decider-2b, laya, sagemaker (any sagemaker:// address). These are stable policy ids, pinned in a test. "No model" is never asked. An endpoint the owner writes by hand is the owner's own and is not governed by the list (maintainer decision, 2026-11-12); the full off switch (capabilities.decisions.enabled: false) and the existing endpoints host list still apply to it as on main. No list means every model is allowed.

Enforcement. capability.provider_id(endpoint, model) names the configured model; capability.is_decisions_denied(provider=...) asks providers:<id> together with the row(s) inside one policy-generation bracket (fails closed if the policy moved between the reads). It is asked at every chokepoint that reads the row for the configured provider:

  • send path: gate._consented_for, so decide, is_enabled and warm refuse a refused model whichever writer named it in config.json;
  • PUT /api/decisions/provider (switch_refused): 403 decisions_capability_denied, nothing written;
  • PUT /api/decisions/consent: an enabling or scope-granting write for a refused model is 403; a disabling write is never refused;
  • the consent carry to a local preset, the SageMaker connection test, and local_runtime.resume_configured (a refused preset is not started).

A refusal never touches the keystone: the recorded consent stays in place and inert, and counts again if a later policy allows that model.

Exact checks, no host parsing. jev is the configured address equal to the hosted default endpoint, a preset is the address the provider route builds for it, sagemaker is the sagemaker:// scheme. Nothing parses or pattern-matches a host, and there is no regex in the new code.

Live policy changes. A governance-generation change on the slots frame now invalidates the provider and consent reads as well as the dashboard config (hooks/websocket/slotList.ts), so an open card follows a policy that allows or refuses a model.

Settings read. GET /api/decisions/provider adds provider_permitted: {id: bool}, each the AND of the model's row, the endpoints list for a model that leaves the machine, and its own entry (dashboard/handlers/decisions_offers.py, asked without a SEL row per id since it is an offer; every request that acts is audited). hosted_permitted / sagemaker_permitted / configured_refused fold the list in, so a configured hosted default the list refuses reads as "No model" with no config rewrite.

UI.

  • Off switch: the card draws its name and one plain notice, "Your administrator's policy turns off decision models for this dashboard, so their settings are not shown here.", with no switch and no picker (role="status"), instead of vanishing. It speaks about this dashboard only and claims nothing about what is sent or runs elsewhere, since a policy may still permit Slack or another channel under a standing consent. Controls the card has drawn stay mounted, hidden, while the policy withholds them, so a typed key (or a SageMaker endpoint, or a budget) is there again when the policy lifts; a failed refetch of the policy shows the read-failure notice, not the last denial.
  • A configured local preset the policy refuses: the card says so FIRST, right under the switch ("Your administrator does not allow the decision model saved in your settings.", and with a consent recorded "... Your agreement to send to it stays recorded until you turn this switch off.", so an "on" switch is not read as the model running), keeps its "Sent to" line naming the address, drops the share line, holds switching on and keeps a recorded consent revocable. With no consent recorded the egress sentence is dropped too; with one it stays, since the refusal is the dashboard surface's answer and a surface the ceiling permits still sends under it. GET /api/decisions/consent reports that refusal as withdrawn, so a consent recorded for the refused address shows the refusal alone, never the "you agreed to a different address, turn this off and on again" warning, whose enabling write would be refused. When every listed model is refused the card drops its "pick one below" line, and the picker says "Your administrator allows none of the listed decision models.", which never claims nothing is sent (a hand-written endpoint may still be). The picker's switch button names what it writes ("Use Laya", "Use no model"). Over a refused model still saved in the settings (the hosted default, or a SageMaker endpoint), no option is drawn selected and nothing is "In use"; the saved model is tagged "Saved, not allowed", and its entry says "{{name}} is still saved in your settings, but your administrator does not allow it. Clear it to stop using decision models, or pick another model below." with a Clear from settings button that writes "No model". Picking "No model" instead offers the usual Use no model.
  • Allow list: every refused model is drawn with its name and figures faded, unselectable, and the line "Not allowed by your administrator." at full contrast; the hosted provider and SageMaker are no longer hidden. When every option is refused the picker says so above them; "No model" is always choosable. No refusal line says decisions fall back to the built-in rules. A refused consent write says which case it is: "decision models are turned off by your administrator" for the off switch, "this decision model is not allowed by your administrator" when the list refuses only the configured model.
  • The hosted provider's entry now holds its own API key field (DecisionsHostedKeyField.tsx), drawn while that entry is chosen; the shared block keeps only the shared settings. A settings-search hit on the key opens nothing: it lands on the hosted provider's entry and highlights it (the registry entry in settingsManual.ts anchors the id on the entry, so the highlight waits for it behind the lazy picker), and the reader opens the entry, and so the key, by choosing it. The flash is a dashed neutral outline (data-setting-flash="neutral"), not the accent ring a selected option wears. Once drawn, the field stays mounted (hidden) while another model is picked, the policy starts refusing the hosted provider, or a re-read fails, so a typed, unsaved key is not lost. The field reports a failed vault read itself ("Could not load secrets: ...") and offers no Save until the read succeeds.
  • New and changed catalog keys (decisions_admin_disabled, decisions_all_models_blocked, decisions_configured_blocked, decisions_provider_blocked, decisions_hosted_refused, decisions_configured_blocked_consent, decisions_provider_clear_saved, decisions_provider_saved_refused, decisions_provider_downloaded) ship in all 12 catalogs, verified with i18n-translate.mjs verify.

Specs and docs, same commit. docs/system-specs/modules/decisions.md (section 7, new 7.2, 7.1), governance.md (the providers list), and the user doc src/kiro_crew/docs/decisions.md (new "When your administrator limits decision models" section). No entry in docs/decisions/ covers these placements.

What the internal edition sets. In the edition companion's bundled security_policy.json:

  • key: capabilities.decisions.scopes.providers, value: {"mode": "deny", "deny": ["jev"]} (the hosted provider the picker offers, and a config naming the exact hosted default endpoint)
  • key: capabilities.decisions.scopes.endpoints, value: {"mode": "deny", "deny": ["api.typesafe.ai"]} (the hosted host in any spelling: only the exact hosted address counts as jev, so a hand-written near copy is refused by the endpoints host list, which the send path asks for every address)
{"version": 1, "capabilities": {"decisions": {"scopes": {"providers": {"mode": "deny", "deny": ["jev"]}, "endpoints": {"mode": "deny", "deny": ["api.typesafe.ai"]}}}}}

If the bundled policy already names capabilities.decisions, merge scopes.providers and scopes.endpoints into that object; the full off switch is capabilities.decisions.enabled = false. A governance profile bound to a surface can carry the same scopes.providers object to narrow that surface only. test_decisions_provider_allowlist.py::TestTheEditionSnippet pins this fragment.

Backwards compatibility

Compatible: no policy, or a policy without the providers list, allows every model, so every saved provider selection, consent and config keeps working; no config key, API field, ceiling key or settings-search id is renamed or removed (provider_permitted and the consent read's withdrawn are added; the settings-search entry for the hosted key keeps its id and moves its anchor, below).

The diff tightens what a policy can refuse and changes what the card draws under a ceiling, so its readers:

Reader: src/kiro_crew/decisions/gate.py:_consented_for -- chat -- test_decisions_provider_allowlist.py::TestTheGateHonoursTheList::test_no_policy_sends_to_every_saved_selection
Reader: src/kiro_crew/decisions/gate.py:_consented_for -- cron -- the autonudge judge reads the same consent; test_decisions_judge_llm.py passes unchanged
Reader: src/kiro_crew/decisions/gate.py:_consented_for -- subagent -- browser.step and spawned-task routing funnel through the same read; test_browser_reflex.py passes unchanged
Reader: src/kiro_crew/dashboard/handlers/decisions.py:_carry_consent -- chat -- test_decisions_local_provider.py::TestProviderRoutePut::test_two_switches_never_interleave_write_and_carry
Reader: src/kiro_crew/dashboard/handlers/decisions_sagemaker.py:switch_refused -- chat -- test_decisions_sagemaker_provider.py::test_a_fleet_denial_of_the_hosted_side_refuses_it
Reader: src/kiro_crew/dashboard/handlers/decisions.py:_provider_payload -- chat -- with no list the side answers are unchanged; test_decisions_sagemaker_provider.py::TestRefusedOptionIsNotOffered
Reader: src/kiro_crew/decisions/local_runtime.py:resume_configured -- release -- runs at gateway start; test_decisions_provider_allowlist.py::TestTheRuntimeResume, test_decisions_local_runtime.py
Reader: website/src/pages/settings/DecisionsCard.tsx:DecisionsCard -- chat -- a denied row draws the notice instead of nothing; decisionsCard.test.tsx "says the administrator turned decision models off, and offers no control"
Reader: website/src/pages/settings/DecisionsProviderPicker.tsx:DecisionsProviderPicker -- chat -- refused options shown greyed instead of hidden; decisionsProviderPicker.test.tsx
Reader: website/src/pages/settings/DecisionsHostedKeyField.tsx:DecisionsHostedKeyField -- chat -- the hosted API key field moves from the card's shared block into the hosted provider's entry, drawn while that entry is chosen; decisionsProviderPicker.test.tsx "draws the key field inside the hosted entry while that entry is chosen, and nowhere else"
Reader: website/src/components/commandPalette/settingsManual.ts:developer.jev-api-key -- chat -- the settings-search id is unchanged, its anchor moves from the key field to the hosted provider's entry, which the hit highlights without opening; decisionsHostedKeyHighlight.test.tsx

Tests

  • test/test_decisions_provider_allowlist.py (new, real policy and evaluator): the catalog row; provider_id per address shape; at the gate, no policy and a list-less policy send, a refused model (deny, allow-other, empty allow) sends nothing and leaves the keystone byte-identical, consent counts again once allowed, a local preset answers to its own entry, the off switch refuses all, a policy swap between row and entry denies, "none" is never asked; the provider route (per-model GET, 403 and no write for a refused hosted provider, preset or SageMaker, "none" never refused); the consent route (403 for a refused model, disabling allowed, GET reports permits: false with consent shown as recorded); runtime resume; the edition snippet.
  • Mutation-tested: dropping the provider from gate._consented_for, switch_refused, _seam_withdrawn or resume_configured, forcing permitted_providers true, removing the generation bracket, and mis-classifying SageMaker each fail that file.
  • Frontend: decisionsCard.test.tsx (the administrator notice and no control; the credential tests now open the hosted entry; a refused SageMaker address), decisionsProviderPicker.test.tsx (refused options greyed, not hidden; the key field lives inside the hosted entry only, is not drawn for a refused provider), previewSurfaces.test.tsx (the card's button census).
  • Regressions for the review findings, each failing on the previous code (mutation-checked): decisionsProviderPicker.test.tsx "keeps a typed, unsaved key while another model is looked at".
  • Round five, each failing on the previous code (mutation-checked): decisionsCard.test.tsx keeps "Sent to" with the refusal after it; decisionsProviderPicker.test.tsx "selects nothing, marks the saved model as saved and refused, and offers to clear it" and "keeps a typed key when the policy starts refusing the hosted provider".
  • Round six, each failing on the previous code (mutation-checked): decisionsCard.test.tsx "keeps a recorded consent revocable under the refusal" shows the refusal and no moved-address warning, decisionsPreview.test.ts reads withdrawn, and test_decisions_provider_allowlist.py / test_decisions_consent.py pin the field (true under a refusal, false for a moved address); decisionsHostedKeyHighlight.test.tsx (a search hit highlights the hosted entry and opens and chooses nothing) and the picker's one-anchor test. The search auto-open, its latch and its tests are gone. useWebSocket.frameTrace.test.ts records the two extra invalidations on a governance generation change (the Frontend Tests failure on the previous head).
  • Round seven, each failing on the previous code (mutation-checked): TestAHandWrittenEndpointIsTheOwners (no list id for a hand-written endpoint, a near copy of the hosted address and a SageMaker runtime URL included; no list, allow-none, allow-jev or deny-every-listed, stops it on the send path, the consent PUT or the provider read; the off switch still does); the exact hosted default is the hosted entry; the stable ids pinned literally; the two 403 sentences; decisionsCard.test.tsx (a hand-written endpoint is never shown refused, even with the local row withdrawn; the egress stays disclosed under a refusal with a recorded consent; the listed-models banner text). The host classifier, impl_sagemaker.is_runtime_host and their tests are deleted. The key-field waits in decisionsProviderPicker.test.tsx carry a named ceiling (HOSTED_KEY_FIELD_DRAWS).
  • Round eight, each failing on the previous code (mutation-checked): decisionsCard.test.tsx "keeps a typed key through a live withdrawal and gives it back when the policy lifts", "shows a failed refresh of the policy, not the last denial as if it were current", and the notice text (about this dashboard only); decisionsProviderPicker.test.tsx names the drawn button in the ready note, gives no step for a model already configured, and gives the administrator's line instead of steps for a refused model, ready or being prepared; TestTheEditionSnippet refuses the hosted host in three spellings and leaves an address of the owner's alone. capture-decisions-local-runtime.mjs drives the renamed button.
  • Round nine, each failing on the previous code (mutation-checked): test_decisions_provider_allowlist.py "the judge llm lane stays on under a refused model" (GET and PUT; the off switch still turns it off); decisionsProviderPicker.test.tsx "selects nothing, marks the saved model as saved and refused, and offers to clear it" and "clears a refused hosted default from the settings, from inside its entry"; decisionsCard.test.tsx the refusal comes first, before the egress sentence and "Sent to", the consent wording, and no share line.
  • docs/feature-map/README.md gains the "Decision model allow list" and "Hosted provider API key" rows.
  • Round three, each failing on the previous code (mutation-checked): near copies of the hosted address under a jev deny on the send path, the consent PUT, the provider read and switch_refused (TestNearCopiesOfTheHostedAddress); the consent read reports no point on for a refused model; useWebSocketGitlabHosts.test.ts "re-reads the decision-model answers on a governance-ceiling generation change"; decisionsCard.test.tsx "a configured model the administrator does not allow" (three cases); the named switch button in decisionsProviderPicker.test.tsx. The credential waits in decisionsCard.test.tsx carry a named ceiling (HOSTED_KEY_DRAWS) over the lazy chain.
  • Lint gates run locally: pinned black gate, isort 6.0.0, flake8 7.1.0, pinned mypy 1.14.1 --platform linux, comment-history, brand-name, file line cap, harness parity, flake-pattern ratchet, the code-review rule checks, npm run i18n:check with the merge base, tsc -p tsconfig.app.json, eslint on touched files, jscpd.

Manual verification

Ran the real built SPA through the screenshot harness (website/scripts/capture-decision-model-governance.mjs, shared serve-dist and stub-dashboard-api helpers) against this branch and against a build of origin/main (--before), with every /api/** call answered from fixtures; the after run asserts each state's text. No gateway or live provider was used.

Screenshots / video

Each pair shows main first, then this branch. Light theme.

Turned off by the administrator, before (no card at all, the page ends at the group above) and after:

before: turned off
after: turned off

Allow list with one model (Laya), before (the policy cannot express it; the hosted provider and SageMaker were hidden when refused) and after:

before: allow list of one
after: allow list of one

All models allowed with the hosted entry chosen (its API key inside the entry), SageMaker active, a local model active

before: all allowed
after: all allowed
before: SageMaker active
after: SageMaker active
before: local model active
after: local model active

The three refusal states this PR adds (after only; main has none of them):

Every listed model refused by the allow list: every option greyed except "No model", with the sentence about the listed models above them:

after: every listed model refused

A configured local model the list refuses, with a consent recorded for it: the refusal first, saying the switch is the recorded agreement, the egress still said, no share line, "Sent to" naming the address, no "agreed to a different address" warning, and the switch still revocable:

after: configured model refused, consent on

A refused hosted default still saved: no option selected, the hosted entry tagged "Saved, not allowed", and inside it the sentence saying it is still saved and what clearing does, with Clear from settings:

after: hosted default refused

A settings-search hit on the hosted provider's key while a local model is in use: the hosted entry is highlighted (a dashed neutral flash, unlike the selected option's ring), nothing opens, the model in use stays selected, and no switch is queued:

after: search highlights the hosted entry

Related Issues

Supersedes #36267 (split for the review lanes' diff limit). Maintainer request (Bolin Chen, bolichen97), 2026-11-12.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributed under the project's Apache-2.0 license.

@bolichen97
bolichen97 requested review from a team and KiroCrewCaptain as code owners November 13, 2026 01:36
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 13, 2026
@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of fda7148dceff8ec1228903b67996a2f2f7b9cc5e — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: the change adds one inner list to the existing capabilities.decisions row, every chokepoint asks it, and the old is_decisions_denied callers still work because provider is optional.

Notes

  • The card used to say a policy-off ceiling "gets NO CARD — not a disabled one". This PR deliberately reverses that, and it is the stated goal. No docs/decisions/ entry records the old choice, so nothing needs superseding.
  • The ## Backwards compatibility section falls past the 8000-byte truncation, so the reader list was not checked. No unlisted broken reader was found.

[DESIGN-REVIEWED] fda7148

Description read

[DESCRIPTION-READ] bff31f18d5ce371011a06936f31917ec833f5be19157f76e1f253603ab70f3ae

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of fda7148dceff8ec1228903b67996a2f2f7b9cc5e — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Nothing blocks. Two UI changes ride along with no named user who was failing: the hosted API key field moved, and the picker's button was relabelled.

Notes

  • Item 8 — unjustified move: the only harm offered is "The hosted provider's API key sits in a 'Shared settings' block although only that provider uses it". That is a grouping argument. No report shows anyone failing to find the key. Every hosted user now has to find it in a new place. Subtraction: keep the SecretField block in DecisionsCard.tsx's global panel, delete DecisionsHostedKeyField.tsx (158 lines), and revert the developer.jev-api-key re-anchoring.
  • Item 9 — rides along, relabel: decisions_provider_use ("Use this model") is replaced by decisions_provider_use_named / _use_none in 14 locale files. The Goal does not cover this, and no misclick is named. Subtraction: keep the existing decisions_provider_use key.

What this change ships

Inventory (10 items) — 8 justified

Intent: let an administrator refuse decision models, all of them or one at a time, and have the gateway enforce it and the dashboard explain it. ADDITION.

  1. Policy can allow or refuse each decision model (capabilities.decisions.scopes.providers) — justified
  2. A refused model sends nothing on decide/warm, whoever wrote the config — justified
  3. Provider and consent writes for a refused model return 403 — justified
  4. A refused local preset is not resumed; the SageMaker test is refused — justified
  5. The off switch now shows an administrator notice instead of hiding the card — justified
  6. Refused hosted/SageMaker options are shown faded as "Not allowed" instead of hidden — justified
  7. A saved-but-refused model is tagged and gets "Clear from settings" — justified
  8. Hosted API key moved from the shared block into the hosted picker entry — unjustified move
  9. Picker button relabelled "Use this model" → "Use Laya" / "Use no model" — rides along
  10. A policy change refreshes the open card's provider and consent reads — justified

[FIRST-PRINCIPLES-REVIEWED] fda7148

Description read

[DESCRIPTION-READ] 9f7a9f249abdc6552ecea3c03daae3b255ec7278f6fbe7974dfd554d00cd10a7

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ✅ PASS (a BLOCK with no Trigger and no Outcome does not block)

UX-level review of fda7148dceff8ec1228903b67996a2f2f7b9cc5e — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

UX-Verdict: PASS

cannot evaluate: missing the picker's renamed save button ("Use {{name}}" / "Use no model", which replaces "Use this model") in the state where a pick is chosen but not yet saved. None of the 14 screenshots shows it.

Evidence gaps

  • Save button after an unsaved pick: the needsSave branch in DecisionsProviderPicker.tsx:741-755 draws the new useLabel button and the reworded decisions_provider_pending line ("…keeps answering until you press the button below."). The blind read never mentions either one. To close: a screenshot with a different model picked but not saved, showing the "Use Laya" button and the pending line.
  • "Use no model" over a refused saved model: the description promises that picking "No model" there offers the usual Use no model. shot-13 shows only Clear from settings. To close: a screenshot of shot-13's state with "No model" picked.

Notes

  • Grey box under a refused model (shot-12): the decisions_configured_blocked line sits right above decisions_provider_memory_cost ("While it runs, Laya keeps about 6GB… Pick No model to free it."). The blind reader said: "no idea. Is Laya running or not?" Hide the memory line when blockedId(chosenPreset.id) is true.
  • decisions_hosted_refused: "Clear it to stop using decision models" sounds bigger than clearing one saved choice; the reader said it "sounds bigger than forgetting one choice". Suggested wording: "Clear it, or pick another model below."

[UX-REVIEWED] fda7148

Description read

[DESCRIPTION-READ] 634ba07f27b05278c5cc22ac064e6f563ca54a0ac5ee40d42b02b6505ce311b6

That sha256 names everything this verdict read: the PR description, plus the 17 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed fda7148dceff8ec1228903b67996a2f2f7b9cc5e — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] fda7148

Verdict parsed from the review's SHA-scoped output markers for commit fda7148dceff8ec1228903b67996a2f2f7b9cc5e.

False positive or not applicable? A repository writer can comment:
/ai-review override fable fda7148dceff8ec1228903b67996a2f2f7b9cc5e false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of fda7148dceff8ec1228903b67996a2f2f7b9cc5e and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] fda7148

False positive or not applicable? A repository writer can comment:
/ai-review override gpt fda7148dceff8ec1228903b67996a2f2f7b9cc5e false-positive: <reason, 20 to 500 characters>

@bolichen97
bolichen97 enabled auto-merge (squash) November 13, 2026 01:50
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 13, 2026
@bolichen97
bolichen97 force-pushed the feat/decision-model-allowlist branch from 3e0efa2 to 9de2c8c Compare November 13, 2026 03:16
@bolichen97

Copy link
Copy Markdown
Contributor Author

Answering the three First Principles notes. Each point is an explicit instruction from the maintainer, Bolin Chen (bolichen97), decided on 2026-11-12; none is a choice this PR made on its own.

  1. The hosted API key moves into the hosted provider's own entry. This is requirement 3: the owner opens a provider and enters its key there, and the key does not sit in a separate section that exists only for that provider. The shared block keeps only the settings every decision model shares.
  2. A per-model providers allow list, with the internal edition refusing jev. These are requirements 2 and 4: every decision model (each local preset, a SageMaker endpoint, a hand-written address, the hosted provider) has its own allow-list entry, enforced in the gateway, rather than only the hosted-host and SageMaker-host ceilings. The edition ships capabilities.decisions.scopes.providers = {"mode": "deny", "deny": ["jev"]} in its bundled policy.
  3. A disabled-by-administrator notice instead of withholding the card. This is requirement 1: when the administrator turns decision models off, the UI says so plainly rather than drawing nothing or a silently greyed switch.

The remaining feature-level "Jev" wording, including the notice's title "Decisions (Jev)" that the UX review pointed out, is the wording pass. It ships in the follow-up PR opened on main once this one merges, so this PR changes only the governance wording it introduces.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Nov 13, 2026
@KiroCrewCaptain

KiroCrewCaptain commented Nov 13, 2026 •

Copy link
Copy Markdown
Collaborator

Code change approver recommendation: A person should decide

Why: the decider thinks the change is probably fine but estimates a 25% chance that this change introduces a defect, above the 12% limit for approving on its own.

I recommend approving. The PR adds a per-model providers allow list as an inner identifier list on the existing capabilities.decisions governance row. The gateway asks for the configured model's entry at every point that already reads the row for that model: the send path, the consent and provider writes, the consent carry, the SageMaker test and the local-runtime resume. All of these use one policy-generation bracket that fails closed, and a refusal leaves the recorded consent untouched. With no list, every model behaves exactly as before. The judge's LLM lane is correctly kept outside the list. GET /api/decisions/provider gains provider_permitted, GET /api/decisions/consent gains withdrawn, and the card replaces the vanished-card state with a plain administrator notice. The new tests run against the real evaluator and cover each of these paths. What remains is the documented maintainer decision to leave hand-written endpoints outside the list, a small reachability regression for the shared API key, and a missing CI run plus a merge conflict. None of these blocks the code.

An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit 300ef2a7e5 · 3362 changed lines. If this recommendation is wrong, reply here: replies are how the approver is evaluated.

@bolichen97
bolichen97 force-pushed the feat/decision-model-allowlist branch 3 times, most recently from b4360f1 to 06f766d Compare November 13, 2026 04:41
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 13, 2026
@bolichen97
bolichen97 force-pushed the feat/decision-model-allowlist branch from 06f766d to 1501dc0 Compare November 13, 2026 05:32
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Nov 13, 2026
@bolichen97
bolichen97 force-pushed the feat/decision-model-allowlist branch from 1501dc0 to 302cca6 Compare November 13, 2026 06:35
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 13, 2026
@bolichen97
bolichen97 force-pushed the feat/decision-model-allowlist branch from 302cca6 to 89d65bb Compare November 13, 2026 08:16
@bolichen97

Copy link
Copy Markdown
Contributor Author

Round 7. Maintainer decision (Bolin Chen, 2026-11-12): "用户自己用custom节点不管我们事情, 我们只要api.typesafe.ai disable就行了, 不用REGEX匹配" (a user's own custom endpoint is not ours to govern; the edition only needs to disable the hosted api.typesafe.ai provider; no regex or host matching).

What that changed on this head:

  • The providers list governs jev, plumb-4b, strands-decider-2b, laya and sagemaker, each by an exact check: jev is the configured address equal to the hosted default endpoint, a preset is the address the provider route builds for it, sagemaker is the sagemaker:// scheme. These are stable policy ids, pinned in test_decisions_provider_allowlist.py.
  • A hand-written endpoint has no list id and is never refused by the list. custom is gone from the id vocabulary, permitted_providers, provider_permitted, the picker and the card. The full off switch still refuses every endpoint, and the endpoints host list is unchanged from main.
  • The host classifier is deleted with its tests: the multi-reading url_policy_hosts matching, impl_sagemaker.is_runtime_host and the FIPS and VPC endpoint spellings. So the round-7 Opus finding (an ANY-reading match letting https://api.typesafe.ai\@elsewhere pass an allow-jev list) and the round-2 near-copy item are moot: no host classification is left. Under this decision a hand-written address is the owner's own whatever host it names.
  • mypy: the CI failure in cli_commands_memory.py:335 is main-red. On origin/main, fix: memory backups --store for an undeclared store prints no traceback #36114 makes declared a list in the backups branch and fix: memory restore --cancel-pending for an undeclared store logs no traceback #36212 makes it a bool in the restore branch of the same function. This branch's base predates fix: memory restore --cancel-pending for an undeclared store logs no traceback #36212, so only the PR merge ref sees it. mypy 1.14.1 from an environment built the CI way (uv pip install -e ".[voice]" --group dev) reports no issues on this branch. That file is untouched here.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Nov 13, 2026
@bolichen97
bolichen97 force-pushed the feat/decision-model-allowlist branch from 89d65bb to ca05283 Compare November 13, 2026 09:16
@bolichen97

Copy link
Copy Markdown
Contributor Author

Round 8, self-review sweep with the lanes' lens before this push. What it found and what changed:

(a) References to a control this PR renamed or removed:

  • decisions_provider_ready and decisions_runtime_preparing_note said "press Use this model". Now: "press the button below" while that button is drawn, a plain "Downloaded." for a model that is already configured, and for a refused model the administrator's line instead of steps (ready or being prepared). All 12 catalogs.
  • src/kiro_crew/docs/decisions.md named Use this model in three places (the SageMaker save, stopping a local model, the local install). Now Use Your SageMaker endpoint, Use no model, and "Use and the model's name".
  • website/scripts/capture-decisions-local-runtime.mjs waited for a "Use this model" button and its ready note. Now it waits for Use Laya and the new note.
  • No other reference to a removed key, the old button, the old clear button, the search auto-open or custom remains in code, catalogs or docs.

(b) Local state and what can unmount it:

  • The hosted key draft, the SageMaker fields, the picked model and the budget box were all lost when a live withdrawal replaced the card with the administrator notice. Now the controls, once drawn, stay mounted and hidden while governance withholds them. Test: type a key, the policy turns off then on, the draft is back.
  • A provider refusal, a failed provider re-read and picking another model were already covered (the key field stays mounted, hidden). The lazy picker load holds no draft yet.

(c) Cached values shown while their refetch fails:

  • The governance denial was shown from cache over a failed refetch. Now a failed refetch shows the card's read-failure notice and holds every write. Test included.
  • The provider read, consent, config, vault index, runtime poll and memory reads already show their own notice and hold writes.

(d) Machine-wide claims:

  • decisions_admin_disabled said nothing is sent and none runs here. Now it speaks about this dashboard only.
  • The user doc said the same in its administrator section, and also "nothing is sent to a model that is not allowed". Both now say where the policy refuses it, since a dashboard-bound profile can leave Slack or another channel permitted under a standing consent.
  • The remaining claims are true of a config-wide or consent-wide action: "No model" means nothing is sent, and a running local model with the switch off is sent nothing.

(e) Loading, error and empty states of the new UI: each has copy. The administrator notice, the listed-models banner, the refusal lines, the key field's vault-read error, the stale provider read, and each preset note.

Also in this round:

  • The spec states, next to providers, that only the exact hosted address counts as jev. A fleet that wants the hosted host refused in any spelling also needs the endpoints host list, and the edition snippet now carries both. TestTheEditionSnippet pins three spellings refused, and an address of the owner's left alone.
  • The Backwards compatibility Reader list now has the hosted-key move and the settings-search entry whose anchor moved.

@github-actions github-actions Bot added the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Nov 13, 2026
@bolichen97
bolichen97 force-pushed the feat/decision-model-allowlist branch from ca05283 to 300ef2a Compare November 13, 2026 10:16
@bolichen97

Copy link
Copy Markdown
Contributor Author

Round 9 self-review sweep, with the lanes' lens.

Docs lint. This PR adds or renames no doc; it changes four existing ones (docs/feature-map/README.md, docs/system-specs/modules/decisions.md, docs/system-specs/modules/governance.md, src/kiro_crew/docs/decisions.md), all already indexed. scripts/docs-lint.sh passes on this branch. The one unreachable doc in run 43400185861 was docs/request-for-change/rfc-app-agent-tools-third-party-grant.md, added on main by #35966 and indexed on main by #36504 after that run, so the next merge ref carries the fix.

Custom-address refusal from before round 7. Fixed in the feature map, the spec, the API client comment, the harness frames and the PR body. Nothing else in specs, user docs, catalogs, en.context.json, comments or test names still describes refusing a hand-written address or a custom list id.

Every status or label derived from a denial, and the refusal it actually obeys:

Status or label Reads The behaviour obeys Match
consent permits, withdrawn; every point row except the judge row + configured model's providers entry the send path gate._consented_for (the same) yes
judge row, LLM lane was row + entry, now the row alone gate._judge_authority LLM lane (row alone) fixed this round
judge row, Jev lane permits (row + entry) the Jev lane goes through _consented_for yes
decisions_enabled (card notice) the rows the off switch yes
hosted_permitted, sagemaker_permitted, provider_permitted, configured_refused, active: none row + endpoints + entry the provider PUT switch_refused, the send path yes
consent PUT 403 and its message row + entry; the message asks the rows alone the same yes
card consentOn (In use, running-off note), scope offers, budget hold withdrawn or moved address nothing sent for the configured model yes

Every state of the card (one combination per row):

State First line or notice Selected radio Tag Button Egress, share, "Sent to"
Off switch notice about this dashboard; no controls none none none none
Off switch after controls were drawn notice; controls kept, hidden hidden hidden hidden hidden
Policy refetch failed read-failure notice; card faded last read last read all held last read
List refuses the saved hosted default, no consent in its entry: "still saved ... Clear it to stop using decision models, or pick another model below." none "Saved, not allowed" Clear from settings egress hidden, no "Sent to", switch-on held
same, consent recorded same none same same; switch revocable egress kept (dashboard-bound)
List refuses the saved SageMaker endpoint the same sentence in the SageMaker entry none on SageMaker Clear from settings as above
List refuses the configured local preset, no consent first: "does not allow the decision model saved in your settings." the preset, greyed "Not allowed by your administrator." none; switch-on held no egress, no share, "Sent to" kept
same, consent recorded first: the same, plus "Your agreement ... stays recorded until you turn this switch off." the preset same switch revocable egress kept, no share, "Sent to" kept
List refuses other models only none the configured one, In use refused ones greyed Use on a pick as normal
List refuses every listed model, No model configured "allows none of the listed decision models." No model, In use all greyed none no egress; switch-on held
List refuses every listed model, hand-written address configured the same banner none (not an option) all greyed Use no model on a pick egress, share, "Sent to", config pointer: it still sends
Hand-written address, no list none none none Use on a pick egress, share, "Sent to", config pointer
No model, no list "No decision model is chosen. Pick one below to turn this on." No model, In use none Use on a pick switch-on held

Governance over decision models knew two special cases, the hosted host
and the SageMaker host. Every decision model now has its own entry in an
allow list, enforced in the gateway, and the dashboard says plainly what
the administrator does not allow.

- capabilities.decisions gains a `providers` inner list (SCOPE_CATALOG
  data, no evaluator change). Ids are the picker's own: jev, each local
  preset, sagemaker, custom. The configured model's entry is asked with
  the row in one policy-generation bracket at every check that reads the
  row for it: the send path, the consent and provider writes, the
  consent carry, the SageMaker test and the runtime resume. A refused
  model is neither selectable (403) nor used, whoever wrote config.json;
  its recorded consent stays in place and inert. No policy, or a policy
  without the list, allows every model, so saved selections keep working.
- GET /api/decisions/provider reports provider_permitted per model. The
  picker draws a refused model greyed with "Not allowed by your
  administrator." instead of hiding it, and the card draws a plain
  "turned off by your administrator" notice when the row is off.
- The hosted provider's entry holds its own API key field; the shared
  block keeps only the settings every model shares.
- Specs (decisions, governance) and the user doc describe the list and
  the notice.
@bolichen97
bolichen97 force-pushed the feat/decision-model-allowlist branch from 300ef2a to fda7148 Compare November 13, 2026 11:29
@github-actions github-actions Bot removed the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Nov 13, 2026
@bolichen97
bolichen97 merged commit a29fcc3 into main Nov 13, 2026
103 of 104 checks passed
@bolichen97
bolichen97 deleted the feat/decision-model-allowlist branch November 13, 2026 12:44
@github-actions github-actions Bot removed the readiness: checking Automated validation is still running label Nov 13, 2026
@adlio adlio mentioned this pull request Nov 13, 2026
5 tasks done
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants