Skip to content

fix: warn when a bot's own file sets a model no session reads - #39466

Merged
KiroCrewCaptain merged 1 commit into
mainfrom
fix/38165-bot-file-model-warning
Nov 19, 2026
Merged

KiroCrewCaptain merged 1 commit into
mainfrom
fix/38165-bot-file-model-warning

Conversation

@bolichen97

@bolichen97 bolichen97 commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A Bot whose own file sets a model says so in the log, so a model that never runs is no longer silent.

A Bot's own file under ~/.kiro/mates/agents/<id>.json can carry a model key: older builds copied the source template into it verbatim, and the template edit and the Bot capability edit can still write it. No session reads that key on any backend (rfc-mate-owned-agent-config §1/§5): a Bot's chat runs on its Model setting (agents.<name>.model) and its workers on its Worker model setting (agents.<name>.worker_model; empty follows Model), and with neither set, on the default model. Nothing said so. The reporter's three Bots named a model in their own files and every turn ran on a different, 2.2x more expensive model, with no log line and no notice.

Why it matters

Anyone with a Bot file from an older build, or one written by the template or capability edit, believes their Bot runs on the model named in that file. It does not, and they find out only from the bill. A log line naming the Bot, the ignored value and the settings that decide the model lets them see and fix it.

Not a goal

  • A diagnose probe or a dashboard notice for the unread key.
  • Refusing, rewriting or stripping a Bot file (or the writers that put model there: the template PATCH, the Bot capability edit).
  • Change-card routing for a model edit.
  • Making sessions read the file's model, or re-resolving the session model from it.
  • The substitute set_model path.
  • Editor notes or save-time checks on the Bot file.

What changed (motivation → approach → change)

A Bot whose own file names a model now gets one warning in the log saying that model is not used, and which settings are.

  • Symptom: a model in a Bot's own file never runs, and nothing says so.
  • Cause: no session reads that key. mate_worker.resolve_worker_spec reads the file on every worker start, on every backend, and dropped the key without a word.
  • Change: right after that read, a model other than empty or auto logs one WARNING per process per Bot and value. It names the Bot, the file and the ignored model, and points to the Bot's Model and Worker model settings. The helper is note_unread_file_model in src/kiro_crew/crewmate_harness.py, kept beside that module's other once-per-process notes and under the same lock, so mate_worker stays stateless.
  • Unchanged: the worker spec and the session model. Nothing is refused, written or re-resolved.
  • Docs: docs/system-specs/modules/crew-mode.md and the mate_worker docstring name the warning where they already say model is ignored.

Backwards compatibility

Compatible: the change only adds a log record. The resolved worker spec, the session model and every refusal are the same as before.

Removes nothing: no path, key, record, validator or public function is hidden, deleted or narrowed.

Tests

  • test/test_mate_worker.py::TestMateFileForMember::test_a_model_in_the_bots_own_file_is_logged_once_as_unread, parametrized:
    • A Bot file with model: "example-model-9" logs exactly one WARNING across two resolves, naming the Bot and that value.
    • A file with "auto", or with no model, logs none.
    • In every case the worker spec carries no model and both resolves are equal.
  • The dedupe set is monkeypatched fresh for each test, and the test uses only tmp_path homes.
  • The test fails on the base (assert 0 == 1) and passes with the fix.
  • Determinism: 20 repeats at -n0 and 10 at -n 4 --dist loadgroup beside two sibling test files all passed, and every test in test/test_mate_worker.py passed in a shuffled order (seed 16726).

Proof: PROVEN test/test_mate_worker.py::TestMateFileForMember::test_a_model_in_the_bots_own_file_is_logged_once_as_unread[example-model-9-1] (head 9c02018)

Manual verification

Run on Linux (Ubuntu 24.04.5 LTS, x86_64) against the real code path, in an isolated home (HOME=$(mktemp -d), KIRO_HOME=$HOME/.kiro, KIROCREW_HOME=$HOME/.kiro/crew). A real config.json binds Bot pixel to its own file $KIRO_HOME/mates/agents/pixel.json, which sets "model": "example-model-9". The script resolves the worker spec twice through resolve_worker_spec and reads resolve_effective_model(cfg, "pixel"). No live model is needed: the defect is the missing log line, not a model call.

Before the fix:

mate_file_for_member: 'pixel'
resolve #1: worker spec model key -> None
resolve #2: worker spec model key -> None
resolve_effective_model(cfg, 'pixel') -> ''
log records mentioning example-model-9: 0

After the fix (same script, plus a happy-path step that rewrites the file with "model": "auto"):

mate_file_for_member: 'pixel'
resolve #1: worker spec model key -> None
resolve #2: worker spec model key -> None
resolve_effective_model(cfg, 'pixel') -> ''
log records mentioning example-model-9: 1
   WARNING kiro_crew.crewmate_harness Bot 'mbr-pixel': its own file 'pixel' sets model 'example-model-9', which is not read; its chat runs on its Model setting and its workers on its Worker model setting. Set the model there.
happy path ('auto'): new 'is not read' records: 0 | prompt merged: True

Real-machine verification

Machine: EC2 m7i.2xlarge, Ubuntu 24.04.5 LTS x86_64, 8 vCPU / 30.8 GiB
Before (main 983fd61, with only this PR's test file applied): python3 -m pytest -n0 -p no:cacheprovider -q <the three test_a_model_in_the_bots_own_file_is_logged_once_as_unread cases> -> 1 failed, 2 passed; [example-model-9-1] fails on assert 0 == 1
After (head 9c02018): the same command -> 3 passed
Cleanup: the throwaway base worktree and the smoke's isolated home were removed; the host is a reused disposable one with no inbound rules and no key pair, terminated by its idle reaper rather than per PR

Related Issues

Closes #38165

Sibling sweep

Swept: git grep -n "load_agent_spec(.*mate_agents_dir|load_agent_spec(mate_dir" -- src/
Sibling: src/kiro_crew/acp/harness/kas.py:297 -- not affected -- resolves a Bot's agent on KAS and never projects model; that Bot's workers start through resolve_worker_spec, which logs
Sibling: src/kiro_crew/mate_worker.py:1203 -- not affected -- this read and the one at :1247 take only tool limits and spec policy fields for a refusal detail or notice, not a worker start; the one at :1421 loads the shared worker base and then resolves the Bot through resolve_worker_spec, which logs
Sibling: src/kiro_crew/mate_capabilities.py:243 -- not affected -- this read and the one at :486 load the shared worker base file, not a Bot's own file
Sibling: src/kiro_crew/mcp_gateway/mate_overlays.py:558 -- not affected -- loads the shared worker base file, not a Bot's own file

Pattern harvest

Not generalizable: the Bot file's model is the one reserved placeholder key that a writer can still set and no reader consumes. The class of a key that is written but never read has no syntactic signature that a lint rule could catch.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributions are accepted under the project's Apache-2.0 license.

@github-actions

github-actions Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 9c02018587b35f184c1bb2d731c2f039dac8f534 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 9c02018587b35f184c1bb2d731c2f039dac8f534 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

This only adds a log line on the one read every worker start already makes. Nothing changes in the spec, the session model or persisted data, and it reverts cleanly.

Notes

  • The warning fires only when resolve_worker_spec runs. A Bot that only ever chats on KAS (acp/harness/kas.py) and never starts a worker stays silent. A follow-up could call the same note_unread_file_model on that read path.

[DESIGN-REVIEWED] 9c02018

Description read

[DESCRIPTION-READ] 73dd327a7f472ee2e6bf050bcbc1814459a0135628ecd45bb07dbe9a5e8b6c6e

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of 9c02018587b35f184c1bb2d731c2f039dac8f534 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Nothing blocks. The fix logs a warning at the one read that drops the key, and its defect is tracked in a linked issue (#38165) with a test that fails on base.

Notes

  • Item 1 — symptom-level: the writers still put the key there ("the template edit and the Bot capability edit can still write it"), so each new file adds another warning. The Goal's ## Not a goal excludes changing the writers, so this is a note, not a demand. Subtraction: stop the template PATCH and the Bot capability edit from copying model into a Bot's own file. The warning would then cover only files from older builds.

What this change ships

Inventory (3 items) — 2 justified

Intent: a Bot owner learns that the model in the Bot's own file never runs, so they set the model where sessions read it. FIX.

  1. A worker start whose Bot file names a model other than empty or auto logs one WARNING per process per Bot and value. The warning names the Model and Worker model settings — symptom-level, the writers still write the key
  2. A new module-level dedupe set _file_model_logged and the helper note_unread_file_model in crewmate_harness — justified
  3. The spec and the docstring now mention the warning where they already say model is ignored — justified

[FIRST-PRINCIPLES-REVIEWED] 9c02018

Description read

[DESCRIPTION-READ] ffdd241f3c5d00c38d8aa31d5a671184d94626a1da087f996807b616a5bb3910

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 9c02018587b35f184c1bb2d731c2f039dac8f534 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 9c02018

Verdict parsed from the review's SHA-scoped output markers for commit 9c02018587b35f184c1bb2d731c2f039dac8f534.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 9c02018587b35f184c1bb2d731c2f039dac8f534 false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 9c02018587b35f184c1bb2d731c2f039dac8f534 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 9c02018

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 9c02018587b35f184c1bb2d731c2f039dac8f534 false-positive: <reason, 20 to 500 characters>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 19, 2026
@bolichen97
bolichen97 force-pushed the fix/38165-bot-file-model-warning branch from 5262ad7 to e026e26 Compare November 19, 2026 16:28
@bolichen97

Copy link
Copy Markdown
Contributor Author

Rebased: main moved in docs/system-specs/modules/ (this PR's spec file, plus 360 files in the directories this PR touches) since this head went green (7505181 -> dc531c1); CI re-running.

The same push fixes the test_only_the_family_imports_an_owner red: mate_worker now reads spec_model through the kiro_crew.agent_discovery facade (e026e26). The test_source_link_github_case red was main's, fixed by #39470, which the new base carries.

[operator: bolichen97]

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Nov 19, 2026
A Bot's own file under ~/.kiro/mates/agents/ may carry a model key
(older builds copied the template into it, and the template and
capability edits can still write it), but no session reads that key
on any backend: a Bot's chat runs on its Model setting and its workers
on its Worker model setting. A model named there silently never ran.

resolve_worker_spec, the one reader of that file on every worker start,
now logs one warning per process per Bot and value when the file's
model is neither empty nor auto, naming the Bot, the ignored model and
the settings that decide it. The worker spec is unchanged; nothing is
refused, written or re-resolved.

Closes #38165

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@bolichen97
bolichen97 force-pushed the fix/38165-bot-file-model-warning branch from e026e26 to 9c02018 Compare November 19, 2026 20:29
@bolichen97

Copy link
Copy Markdown
Contributor Author

Rebased: main moved in this PR's spec file (docs/system-specs/modules/, same file), src/kiro_crew/context_assembly/turn.py (imports kiro_crew.crewmate_harness) and 18 files in the directories this PR touches (38 commits) since this head's CI ran (dc531c1 -> 983fd61); CI re-running.

The previous head's only reds were main's Windows flakes, each fixed on main after that head's base: test_pr_watchers.py::TestCloneLifecycleAndExport (#39545, plus #39516's return to 4 Windows xdist workers) and the code-channel bar-budget test test_the_bar_is_resent_only_when_its_items_change (#39505, plus #39516). The commit itself is unchanged: the same patch on the new base.

[operator: bolichen97]

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Nov 19, 2026

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I reviewed head 9c02018. When a Bot's own file names a model other than empty or auto, the change logs one warning per process per Bot and value. It does not change the resolved worker spec or the session model. I confirmed that neither the session model resolver nor the KAS projection reads the model key from a Bot's own file, so the warning is accurate. The new parametrized test fails on the base and passes here, and the rest of the mate worker suite passes. Nothing blocks.

@KiroCrewCaptain
KiroCrewCaptain merged commit 73758c4 into main Nov 19, 2026
99 checks passed
@KiroCrewCaptain
KiroCrewCaptain deleted the fix/38165-bot-file-model-warning branch November 19, 2026 21:51
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Nov 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

An agent pinning a model that does not exist silently runs on a model costing 2.2x, with no warning

2 participants