Skip to content

fix(acp): record a withheld kiro startup pin as refused - #19024

Merged
iamwhatever merged 1 commit into
kirodotdev:mainfrom
javenciu:fix/acp-record-refused-model-pin
Oct 11, 2026
Merged

iamwhatever merged 1 commit into
kirodotdev:mainfrom
javenciu:fix/acp-record-refused-model-pin

Conversation

@javenciu

@javenciu javenciu commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: when a kiro session withholds a startup model pin that the account does not serve, the pin is recorded in model_pin_refused, so the cron "Model unavailable" note and the usage row reflect the model that actually ran.

  • Both kiro start paths withhold such a pin and leave the session on the backend default. The dedicated runtime does it in AcpClient._apply_startup_model (src/kiro_crew/acp/client.py). The shared runtime does it in the kiro start in src/kiro_crew/providers/acp.py.
  • Neither path recorded the pin. Callers that bill or label a turn by the pin read model_pin_refused through provider_model_pin_refused (src/kiro_crew/llm_helpers.py), so they saw nothing.
  • The cron acquire step in src/kiro_crew/slack/gateway.py returns bool(job.model) and provider_model_pin_refused(client) as its "downgraded" flag. With the field empty, the result gets no "Model unavailable" note and the usage row bills the pinned model.

Why it matters

A user reads a cron result as the pinned model's work, and the cost report bills a model that never ran. kiro is the default backend, so this is the common case of the problem that PR #14971 fixed for config-option backends. That PR listed this path under its Not a goal ("A pin dropped before the push because the advertised list lacks it").

Not a goal

  • Sub-agent and workflow usage rows. src/kiro_crew/subagent_manager/run.py and src/kiro_crew/workflows/agent_exec.py bill the explicit pin and read neither model_pin_refused nor model_pin_partial. PR #14971 names them under its Not a goal, and they stay there.
  • A pin that another harness's model catalog claims (the model_scope branch on both start paths). It is scoped out on purpose and without a warning, because it answers "which harness was this picked in", not "does the account serve it". Whether a cron pin picked under another harness should read as a downgrade is a separate decision.
  • Making the startup withhold raise. It stays non-strict, as PR #14971 kept the config-option push.
  • Any new signal or reader. The existing field and its two readers (the cron acquire step and chat_handlers._try_live_model_switch) carry the change.

What changed (motivation → approach → change)

  • Symptom: a kiro session whose startup pin the account does not serve runs on the backend default, and model_pin_refused stays "".
  • Root cause: only the config-option branch of AcpClient._apply_startup_model wrote model_pin_refused. The kiro entitlement withhold in the same function, and the matching withhold in the shared-runtime start, logged a warning and returned.
  • Change, src/kiro_crew/acp/client.py: the withhold branch of _apply_startup_model sets self.model_pin_refused = self._model before it records the session as running DEFAULT_MODEL. The field's comment names both writers.
  • Change, src/kiro_crew/providers/acp.py: the shared-runtime kiro start sets handle.model_pin_refused = configured_model in its withhold branch. A later AcpSessionHandle.set_model call still resets the field first, so an explicit pick reads only its own verdict.
  • Change, comments and docs: the AcpSessionHandle.model_pin_refused comment (src/kiro_crew/acp/session_handle.py), the provider_model_pin_refused docstring and the model_pin_refused passage of docs/system-specs/common/model-selection.md name the withhold.
  • Rejected: comparing the served model with job.model at the cron call site. The field is the record both start paths and both readers already share. A pin folded to its advertised spelling (openrouter::z-ai/glm-5.3-flash runs as z-ai/glm-5.3-flash) is served under a different id than the pin, so a string comparison would report a downgrade that did not happen.
  • Rejected: raising AcpModelUnavailable at startup. The withhold is deliberate: the pin was not chosen for this turn, and the cron path's except retry would cold-start a second session.

Backwards compatibility

Compatible: only the withhold branch on kiro changes, and only by setting an existing field that every reader already treats as "the pin never ran". A pin the account serves records nothing, as before.
Removes nothing: the change adds two assignments, comment text and one spec paragraph.
Security: no redaction, auth or sandbox rule is touched; the entitlement check that withholds the pin is unchanged, and the change only records its outcome.

Tests

Added:

  • test/test_acp_client.py::TestModelEntitlementPreflight::test_startup_withhold_records_the_pin_as_refused: the dedicated client records the withheld pin.
  • test/test_acp_provider.py::TestStartKiroRuntimeModelEntitlement::test_a_withheld_configured_model_is_recorded_as_refused: the shared-runtime start records it on the handle and never calls set_model.
  • test/test_acp_provider.py::TestStartKiroRuntimeModelEntitlement::test_an_applied_configured_model_records_no_refusal (control): a pin the account serves is applied and records nothing.
  • test/test_acp_client.py::TestModelEntitlementPreflight::test_startup_still_applies_a_usable_model (control) gains assert client.model_pin_refused == "".
  • The fake handle in that class's _run starts with model_pin_refused = "", as a real AcpSessionHandle does.

Before the fix, on origin/main 7bb9d57, this branch's base (both test files copied into a clean checkout, -k "TestModelEntitlementPreflight or TestStartKiroRuntimeModelEntitlement" -n 0, 3 of 3 serial runs). Each failure is a withheld pin left out of model_pin_refused, on the dedicated client and on the shared-runtime handle:

FAILED test/test_acp_client.py::TestModelEntitlementPreflight::test_startup_withhold_records_the_pin_as_refused - AssertionError: assert '' == 'claude-opus-4.8'
FAILED test/test_acp_provider.py::TestStartKiroRuntimeModelEntitlement::test_a_withheld_configured_model_is_recorded_as_refused - AssertionError: assert '' == 'claude-opus-4.8'
2 failed, 25 passed, 768 deselected

After the fix: the same selection gives 27 passed in 3 of 3 serial runs. Every other test file that names model_pin_refused, _apply_startup_model, _start_kiro_runtime or provider_model_pin_refused (21 files, including test/test_cron_mirror_backend_model_pin.py, the cron model-pin tests that use provider_model_pin_refused) gives 1637 passed, 5 skipped, 1 failed. test/test_acp_client.py runs only through the selection above, because its other classes use the real home directory. The failure is test/test_model_served_default.py::TestClientInheritExits::test_withheld_pin_still_lands_on_a_served_model, and this change does not cause it. Run alone against upstream main at 7bb9d57617, this branch's base, it fails the same way: it sends the pinned claude-opus-4.8 where the test expects the served default gpt-5.6-sol. mypy, flake8, isort and black are clean on the changed files. scripts/check_black_formatting.py, scripts/docs-lint.sh and scripts/check_comment_history.py with COMMENT_HISTORY_BASE_REF=origin/main pass.

Manual verification

N/A: the change sets a field on two start paths. The tests drive both paths with the advertised-model shapes the backends report.

Related Issues

None found for this path. Searched PRs and issues for model_pin_refused, pin withhold, not available to this account, model downgrade, served model usage, withheld pin. Related: PR #14971 (config-option backends), PR #18248 (records an explicit set_model refusal on the shared runtime).

Pattern harvest

Rule candidate: review-prompt
Pattern: an outcome record ("this pin never ran") is written on one branch that reaches the outcome and not on a sibling branch that reaches the same outcome.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable): docs/system-specs/common/model-selection.md
  • No secrets, credentials, or internal references in the diff

@javenciu
javenciu requested review from a team and KiroCrewCaptain as code owners October 11, 2026 03:17
@javenciu
javenciu requested a review from buluoray October 11, 2026 03:17
@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: action required A blocking check or review needs attention labels Oct 11, 2026
@github-actions

github-actions Bot commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5, fork) — ⏭️ skipped

Revision 1d972a7e800cff8b831356bc095f85f939877033 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Advisory — does not block merge.

@github-actions

github-actions Bot commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5, fork) — ✅ PASS

Premise-level review of 1d972a7e800cff8b831356bc095f85f939877033 via the fork AI-review pipeline — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

Check the dedicated client: confirm that a later live set_model clears the withheld pin before _try_live_model_switch reads the field (client.py:4915 appears to).

What this change ships

Inventory (5 items) — 5 justified

Intent: when a kiro session runs on the backend default because the account does not serve the pinned model, the cron "Model unavailable" note and the usage row should say so. This is a FIX.

  1. On the dedicated runtime, a withheld startup pin is now recorded in model_pin_refused (client.py _apply_startup_model). — justified
  2. On the shared runtime, the kiro start's withhold branch now records handle.model_pin_refused (providers/acp.py). — justified
  3. As a result, a cron result on a withheld pin gets the downgrade note, and its usage row no longer bills the pinned model. — justified
  4. Comments on both fields, the provider_model_pin_refused docstring and a model-selection.md paragraph now mention the withhold. — justified
  5. Tests: two added tests fail on base, one control test is added, and an existing test gains an assertion. — justified

[FIRST-PRINCIPLES-REVIEWED] 1d972a7

Description read

[DESCRIPTION-READ] 25115b889e87f9238aa1ba6981a24c79bed796fe19b4aa27ed2274720bf8d42b

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5, fork) — ✅ PASS

Design-level review of 1d972a7e800cff8b831356bc095f85f939877033 via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Check that the kiro METHOD_SET_MODEL startup branch, which never clears model_pin_refused, can't carry an earlier withhold forward when a reused client restarts.

[DESIGN-REVIEWED] 1d972a7

Description read

[DESCRIPTION-READ] 7751c103721fcdaa748d1cd64ed83c0fa31ac0056cbeecd6bc6e0bceea17f1ad

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review (fork) — ✅ no blocking findings

Reviewed 1d972a7e800cff8b831356bc095f85f939877033 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 1d972a7

@github-actions

github-actions Bot commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review (fork) — ✅ no blocking findings

Reviewed 1d972a7e800cff8b831356bc095f85f939877033 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 1d972a7

When a kiro session starts with a pinned model the account does not
serve, both start paths withhold the pin and leave the session on the
backend default, with a warning in the log. Neither recorded the pin
in model_pin_refused, the field every caller that bills or labels a
turn by the pin reads through provider_model_pin_refused. The cron
acquire step therefore reported no downgrade: the result carried no
"Model unavailable" note and the usage row billed the pinned model,
which never ran.

AcpClient._apply_startup_model (dedicated runtime) now sets
model_pin_refused to the pin before it records the session as running
the default, and the kiro start in providers/acp.py (shared runtime)
sets it on the session handle in the same withhold branch. The field
comments, the provider_model_pin_refused docstring and the
model_pin_refused passage of the model-selection spec name the
withhold.
@javenciu
javenciu force-pushed the fix/acp-record-refused-model-pin branch from 39e5040 to 1d972a7 Compare October 11, 2026 14:12
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 11, 2026

@iamwhatever iamwhatever left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR records a kiro startup pin that the account does not serve in model_pin_refused, on both start paths (AcpClient._apply_startup_model and the shared-runtime kiro start in providers/acp.py), so the cron result gets its "Model unavailable" note and the usage row stops billing a model that never ran. That is the root cause: only the config-option branch wrote the field, and the entitlement withhold that reaches the same outcome did not. I read the full diff at head 1d972a7 and checked every writer and reader of the field: a later explicit set_model still resets it first on both the client and the handle, so a later pick reads only its own verdict; after a withhold _model is DEFAULT_MODEL, so a re-run of the startup path returns early and leaves no stale refusal for a pin that later ran; and the foreign-scope branch is left alone, as stated. The new tests fail on main and pass on the branch, with controls for a served pin. It changes no entitlement check, only records its outcome. — crew-pr-reviewer (deep re-check)

@iamwhatever
iamwhatever enabled auto-merge (squash) October 11, 2026 18:07
@iamwhatever
iamwhatever merged commit 3b8b223 into kirodotdev:main Oct 11, 2026
104 checks passed
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants