Skip to content

feat(crew): pick a crewmate's agent harness from the runnable list - #29498

Merged
bolichen97 merged 1 commit into
mainfrom
feat/crewmate-harness-picker
Nov 6, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
feat/crewmate-harness-picker

Conversation

@iamwhatever

@iamwhatever iamwhatever commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A person can pick each crewmate's agent harness in the crewmate's profile, choosing only from harnesses that crewmate can actually run on.

Today a crewmate's harness pin (agents.<name>.acp_backend) can only be set by hand in config.json, or through a profile row hidden behind the crewmate-harness developer preview (#25485). Even that row lists every selectable harness, including ones a crewmate cannot run on. Picking one of those stores a pin that refuses later.

Why it matters

People who want one crewmate on Claude Code, and the rest on the default, have no normal way to say so. The preview row lets them store a pin that can never run a crewmate. A list the person can trust, with each harness's limits named, removes that trap.

Not a goal

  • How an unset crewmate resolves its harness. That is unchanged here; docs(rfc): a crewmate's harness follows Settings -> Agent Harness when unset #29442 covers it.
  • Starting a crewmate on a hand-edited pin it cannot run: it still refuses at start, as today.
  • Making Codex or other harnesses run crewmate workers. The picker only names today's limits.
  • A second picker in the crew editor dialog (CrewEditorNextDialog). The profile row stays the one picker.

What changed (motivation → approach → change)

Goal → one server-side list that both the picker and the save use → the existing row out of preview.

  • One list, on the server. crewmate_harness.crewmate_harnesses(config) returns the harnesses that are selectable here (selectable_backends(), which already applies the enterprise agent_backend governance lock) AND crewmate-capable (ACP_BACKENDS_MEMBER_CAPABILITIES ∩ ACP_BACKENDS_MEMBER_DISPATCH). Each one is judged on the runtime its pin starts on (CrewBackendPin.resolve), so a Kiro CLI pin is offered only where kiro-cli runs it on KAS, and a KAS pin is not offered when boot recorded that KAS cannot start here (enterprise_boot.refuse_unstartable_kas). The two worker-limit lists are judged the same way, because a worker checks its session's runtime backend.
  • The readout serves it. GET /api/agents/resolved-model returns it as harnesses, beside the existing worker_harnesses.
  • The save takes only that list. harness_pin_rejected(value, config) checks against crewmate_harnesses(config). Empty still clears the pin.
  • The row leaves preview. The owner said yes to the field. The crewmate-harness preview flag, its Feature Previews card, its i18n keys and its settings-registry entry are removed. The row now shows on every crewmate's Settings page. The "Choose an agent harness" button on the harness-error notice now always shows, because the row it opens is always there.
  • Real limits only. A harness outside ACP_BACKENDS_MATE_WORKER_KIT says its workers cannot start there; a harness with no limit has no line under it.
  • Only runnable harnesses are choices. A stored pin outside the list (goose, opencode, a hand-edited Kiro CLI) is not offered. The Settings row names it as " (unavailable)" and a notice says " is set to , which Bots can't run on here. Choose another agent harness, or Inherited." It makes no claim about what the Bot runs on instead; feat(crew): a Crew bot's harness follows Settings -> Agent Harness when unset #29509 owns that fallback.
  • Models follow the harness. A harness pick reads the new harness's model list. A model or worker-model pin it serves is kept; any other goes back to that harness's default (Inherited). Picking Inherited resets both pins, because the server judges an unpinned Bot's models on a route the card cannot see. If a list cannot be read, the pick stops with an error and nothing is saved.

Product-shape: this moves a control out of a developer preview to every user. The RFC for it is being written in parallel under docs/request-for-change/; it is not on main yet and will be linked here when it exists.

Backwards compatibility

Breaking: PUT /api/agents/<name> with an acp_backend that is selectable but not crewmate-capable now answers 400 invalid_acp_backend instead of storing it. Writer sweep on origin/main @ 405155b: the only writer of acp_backend through this route is CrewProfileSettings.tsx, which now offers only the server list; no other caller sends acp_backend (git grep -n "acp_backend" -- website/src src/kiro_crew/dashboard).

Reader: src/kiro_crew/dashboard/agent_admin/crew_update.py:api_kirocrew_agent_update -- crew page -- the one caller of harness_pin_rejected; updated, test test_a_selectable_but_uncapable_harness_is_refused
Reader: website/src/pages/members/CrewProfileSettings.tsx:CrewProfileSettings -- crew page -- the one reader of PREVIEW_CREWMATE_HARNESS; the row no longer reads it, test reads Inherited and names what that runs on, with no preview switch
Reader: website/src/pages/settings/FeaturePreviewsSection.tsx:FeaturePreviewsSection -- crew page -- the preview card is removed with the flag; settings tests pass
Reader: src/kiro_crew/crewmate_harness.py:CrewBackendPin.resolve -- chat|cron|subagent -- start path unchanged, so an already-stored pin keeps the behaviour it has today; test test_a_hand_edited_uncapable_pin_stays_named_in_the_readout

Tests

  • test/test_crewmate_harness.py: offered list = selectable ∩ capable; a capable but unselectable harness is not offered; a governance lock removes a harness; a Kiro pin is offered only where it runs on KAS; worker limits name their harnesses; the save accepts exactly the offered list; the API refuses an uncapable harness; the readout keeps naming a hand-edited pin.
  • website/src/pages/members/CrewProfileSettings.test.tsx: no preview switch; offers exactly the server list plus Inherited; Inherited clears the pin; only real limits show under a harness; a stored pin outside the list is not a choice and the notice names it; a harness pick keeps a model the new harness serves and resets one it does not; Inherited resets both pins; a failed model list read saves nothing; the keyboard can reach the choices when no option is checked.

Proof: not provable -- other: feature addition, not a fix; the new tests lock in new behaviour

Manual verification

Rendered the real built SPA with every /api/** stubbed (no gateway), and took the shots below. Local GPT and Opus reviews found nothing.

Screenshots / video

The Settings page: the Agent harness row now shows with no preview switch.

Crewmate Settings page with the Agent harness row, dark

The harness page: only the server's list. A harness with no limit has no line under it.

Agent harness choices with limits, light

A stored pin the list does not offer (goose): the Settings row reads "goose (unavailable)", the notice names it, and the harness page lists only runnable choices.

Settings row with an unavailable stored pin

The harness page with a stored pin outside the list

A pin that can't start here: the notice and its "Choose an agent harness" button, which now always shows.

Harness error notice with its button

Other theme

Crewmate Settings page, light

Agent harness choices, dark

Related Issues

Part of #21910

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributions are accepted under the project's Apache-2.0 license; see the Licensing section of CONTRIBUTING.md.

@iamwhatever
iamwhatever requested review from a team and KiroCrewCaptain as code owners November 4, 2026 08:28
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 4, 2026
@github-actions

github-actions Bot commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ✅ PASS (a BLOCK with no Trigger and no Outcome does not block)

UX-level review of 0e1cb6dac21b5dd680e7f13be8cb8d9e1e88bb73 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

UX-Verdict: PASS

cannot evaluate: missing the harness page as this commit renders it. Every harness-page screenshot shows a caption the diff deletes, and no shot shows the per-option model-effect lines that replace it.

Evidence gaps

  • Harness page caption. Shots 02, 04 and 07 show "…keeps Model and Worker model if the new harness offers them, else sets them back to Inherited." The diff removes caption: t('pages.membersPage.profile_harness_hint', …), and profile_harness_hint no longer exists at HEAD. These images are from another revision. Re-shoot the harness page from HEAD.
  • Per-option effect lines. No shot shows "Keeps {{model}}." / "Model goes back to Inherited." / "Keeps worker model {{model}}." / "Worker model goes back to Same as {{name}}." under an option. This is the new way a user learns what a pick does. Attach a shot of a Bot with Model and Worker model pinned, with the harness page open.
  • Model-list error notice. No shot shows profile_harness_models_failed ("Couldn't read the model list for {{harnesses}}…"). Attach a shot of that state.

Notes

  • The reader couldn't tell "Inherited (Kiro Agent)" from the "Kiro Agent" option. Make the Inherited hint say what it adds over the direct pick, for example that it follows the crew default if that default changes.
  • profile_harness_not_supported says "which Bots can't run on here". The reader had "no idea what 'Bots' means". Use the Bot's name, as profile_harness_unavailable already does: "{{name}} can't run on {{harness}} here."
  • On shot 04 nothing is ticked and the notice has no button. Add a line at the top of the list that names the stored pin, for example "Current: goose (unavailable)", so the user can see what is selected now.

[UX-REVIEWED] 0e1cb6d

Description read

[DESCRIPTION-READ] 29f6a301da77c9b62f5ae05c674fb7fd7fa5ddd9583332417570a51663a3dc06

That sha256 names everything this verdict read: the PR description, plus the 10 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 0e1cb6dac21b5dd680e7f13be8cb8d9e1e88bb73 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks. The picker and the save now check the same server list. Removing the preview flag is safe, because the loader drops unknown feature_previews keys rather than refusing them.

Notes

  • crewmate_harnesses(config) probes every selectable backend's pin, and runs refuse_unstartable_kas, on each readout and each save. Keep those calls cheap and free of side effects, or cache them per config load.

[DESIGN-REVIEWED] 0e1cb6d

Description read

[DESCRIPTION-READ] 5048a77eff52c57d10236b1772fc6602a08ec55ea7023cf55d0b9f5794b6fb4a

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ human override accepted

@iamwhatever overrode this lane for 0e1cb6dac21b5dd680e7f13be8cb8d9e1e88bb73 via /ai-review override (this commit only).

@github-actions

github-actions Bot commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 0e1cb6dac21b5dd680e7f13be8cb8d9e1e88bb73 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 0e1cb6d

Verdict parsed from the review's SHA-scoped output markers for commit 0e1cb6dac21b5dd680e7f13be8cb8d9e1e88bb73.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 0e1cb6dac21b5dd680e7f13be8cb8d9e1e88bb73 false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings (all downgraded on adjudication)

GPT 6.1 flagged blocking issues on 0e1cb6dac21b5dd680e7f13be8cb8d9e1e88bb73; Opus 5.5 adjudication downgraded every one of them. Downgraded: the remedy each one needs is disproportionate to the harm it prevents. Adjudication downgraded all 1 blocking finding(s): the remedy each one requires is disproportionate to the harm it prevents.

This comment is updated in place on each push.

Review details

BLOCKING -- website/src/pages/members/CrewProfileSettings.tsx:156 -- Confirmed Auto-only entitlement prevents runnable harness changes

if (!names.some((n) => n !== 'auto')) return null
Trigger: A Bot with a non-Auto model pin switches from Claude Code to KAS through kiro-cli while a live KAS session confirms an Auto-only entitlement. /api/models?backend=kas returns that entitlement, but harnessServes treats it as cold and save retains the unsupported pin.
Outcome: api_kirocrew_agent_update revalidates the retained pin against KAS and returns 400 invalid_model or invalid_worker_model; the runnable harness choice cannot be saved instead of resetting unsupported pins.
Anchor: defect
Fix: Distinguish unknown catalogs from confirmed Auto-only catalogs, and reset unsupported pins for the latter while preserving them for genuinely unknown catalogs.
Sites: website/src/pages/members/CrewProfileSettings.tsx:253 preserves both pins on this result; website/src/pages/members/CrewProfileSettings.tsx:352 repeats the Auto-only inference in the option descriptions.
[BLOCK-MERGE-DOWNGRADED] 0e1cb6d
[GPT-REVIEWED] 0e1cb6d

Adjudication (Opus 5.5) — is blocking on each finding proportionate?

F1 — LOW. I confirmed every condition in code: CrewProfileSettings.tsx:156 returns null when the list holds only auto, :253 then keeps both pins, and crew_update.py:406-416/:426-435 refuse the kept pin with a 400. The 400 isn't silent. CrewProfileSettings.tsx:297-299 puts the error on screen and resets the draft, and nothing is written. The user can get out by setting the model to Inherited and switching again. It also needs a rare combination: a non-Auto pin, a harness switch, and a live entitlement that offers only Auto.
The real fix is expensive. /api/models (agents.py:1396) returns a bare list with no way to tell a cold catalog from a confirmed Auto-only one. So the fix needs a new field across the kiro, KAS-bundle and advertised routes, a frontend branch, and a matching fix at :352. That outweighs a visible, recoverable save refusal on a rare path.

[ADJUDICATION] 0e1cb6dac21b5dd680e7f13be8cb8d9e1e88bb73 total=1 uphold=0 downgrade=1
DOWNGRADE F1 website/src/pages/members/CrewProfileSettings.tsx:156 reason=disproportionate-remedy
[GPT-ADJUDICATED] 0e1cb6dac21b5dd680e7f13be8cb8d9e1e88bb73

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 0e1cb6dac21b5dd680e7f13be8cb8d9e1e88bb73 false-positive: <reason, 20 to 500 characters>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention merge conflict Branch has merge conflicts with its base — author must resolve before merge and removed readiness: checking Automated validation is still running labels Nov 4, 2026
@iamwhatever
iamwhatever force-pushed the feat/crewmate-harness-picker branch from b4510b1 to 713b3e5 Compare November 4, 2026 08:56
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Nov 4, 2026
@iamwhatever
iamwhatever force-pushed the feat/crewmate-harness-picker branch from 713b3e5 to 9443c47 Compare November 4, 2026 09:19
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed merge conflict Branch has merge conflicts with its base — author must resolve before merge readiness: checking Automated validation is still running labels Nov 4, 2026
@iamwhatever
iamwhatever force-pushed the feat/crewmate-harness-picker branch from 9443c47 to 31c91ce Compare November 4, 2026 10:26
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Nov 4, 2026
@iamwhatever
iamwhatever force-pushed the feat/crewmate-harness-picker branch from 31c91ce to 8d62eb1 Compare November 4, 2026 11:57
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 4, 2026
@iamwhatever
iamwhatever force-pushed the feat/crewmate-harness-picker branch from 8d62eb1 to 94471b8 Compare November 4, 2026 13:21
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Nov 4, 2026
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention merge conflict Branch has merge conflicts with its base — author must resolve before merge and removed readiness: checking Automated validation is still running labels Nov 4, 2026
@iamwhatever
iamwhatever force-pushed the feat/crewmate-harness-picker branch from 22072af to 2d2e05b Compare November 5, 2026 04:41
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running merge conflict Branch has merge conflicts with its base — author must resolve before merge labels Nov 5, 2026
@iamwhatever
iamwhatever force-pushed the feat/crewmate-harness-picker branch from 2d2e05b to f3e2a23 Compare November 5, 2026 05:48
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Nov 5, 2026
@iamwhatever
iamwhatever force-pushed the feat/crewmate-harness-picker branch from f3e2a23 to 5a3c426 Compare November 5, 2026 06:48
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Nov 5, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor Author

Disposition for GPT 6.1 F1 ("Auto-only entitlement is mistaken for an unknown catalog", website/src/pages/members/CrewProfileSettings.tsx:154). Head b5f762f64db51607a7e73f9b72028cd8f3467f1c.

Not changed, by owner decision. An auto-only list keeps the pins.

  1. GET /api/models?backend=kas answers a cold catalog (models_kas_bundle._cold_response) and a real auto-only list with the same body: HTTP 200, auto alone. The client cannot tell them apart.
  2. Keeping a pin is the safe side that can be undone. Clearing it deletes the person's choice with no warning, and a cold catalog after a restart is the common case.
  3. A kept pin that is truly unserved is still refused by the server's pin check on PUT /api/agents/{name} (crew_update.py, _model_pin_rejected). The person sees the error and nothing is written.
  4. The picker shows no Keeps / goes-back line for an auto-only list, so it promises nothing.

This matches the previous GPT round on 262be41aff, which found that clearing on a cold auto-only list deletes valid pins ("A cold catalog deletes valid model pins"). A cold flag from the server is out of scope for this PR.

F2 (a failed per-harness model read was silent) is fixed in this head: the picker names the harness whose list failed.

The profile's Agent harness row leaves preview. It offers only the harnesses a
crewmate can run on here (selectable on this gateway and crewmate-capable, so
governance locks apply), computed once on the server by
crewmate_harness.crewmate_harnesses(); a save accepts exactly that list.
Inherited clears the pin. Each choice names its known worker limits, and a
stored pin the list does not offer stays shown, marked unavailable.

Part of #21910
@iamwhatever

Copy link
Copy Markdown
Contributor Author

/ai-review override first-principles 0e1cb6d owner-decision: Owner accepted taking the crewmate harness row out of preview, listing only runnable harnesses, before the RFC status flips from draft; decision recorded in RFC #29442 and amendment #30116 (both merged) and in #29509 (comment)

@github-actions

github-actions Bot commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

Human judgment recorded

@iamwhatever recorded a owner-decision release (class owner-decision) for the first-principles AI finding on 0e1cb6dac21b5dd680e7f13be8cb8d9e1e88bb73.

Owner accepted taking the crewmate harness row out of preview, listing only runnable harnesses, before the RFC status flips from draft; decision recorded in RFC #29442 and amendment #30116 (both merged) and in #29509 (comment)

This decision applies only to this commit. A new push requires a new judgment.

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving 0e1cb6d. I read the product hunks in crewmate_harness.py, crew_update.py, crew_records.py and sections_dashboard.py, and the picker change in CrewProfileSettings.tsx.

  • The resolution order is untouched: a crewmate's pin still wins, then the inherited value, as in the signed #29442 direction. No stored pin is rewritten; a saved kiro pin reads back as kiro.
  • crewmate_harnesses(config) is the one list both the picker and a save use: selectable here (so governance's agent_backend scope still narrows it), able to start here (a KAS that boot proved unstartable is left out), and crewmate-capable (ACP_BACKENDS_MEMBER_CAPABILITIES & ACP_BACKENDS_MEMBER_DISPATCH), judged by the runtime the pin actually starts on. That keeps a save from storing a pin a crewmate cannot run on, which is input validation, not a new security restriction.
  • The picker leaves Feature Previews and shows an "Inherited" choice that clears the pin. Tests cover the list, the governance lock, the Kiro CLI case, the unstartable KAS case and the save refusal.

No new regex. All checks pass on this head (97 ok, 0 red, 0 pending), and no human has an open change request.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants