Skip to content

feat(crew): let a mate pin its own agent harness - #25485

Merged
bolichen97 merged 2 commits into
mainfrom
feat/crewmate-harness-pin
Oct 30, 2026
Merged

bolichen97 merged 2 commits into
mainfrom
feat/crewmate-harness-pin

Conversation

@kyleseaman

@kyleseaman kyleseaman commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A crewmate can run on its own agent harness, with that harness's models, chosen in its profile settings.

Today every crewmate's chat runs on one install-wide setting, agent.member_acp_backend. A crewmate can pin a model and an effort, but not the harness under them, so two crewmates cannot run on different harnesses, and choosing one for a single crewmate moves all of them.

Why it matters

Harnesses differ in model catalog, cost, provider and capability. A developer who wants one crewmate on Claude Code and the rest on KAS has to switch every crewmate today. A per-crewmate pin moves just that crewmate, with a model list that matches what its harness can run, and keeps everything else on the defaults.

Not a goal

  • Pinning a harness for template chats, sub-agents, Slack/channel turns or anything that is not a crewmate's own chat or its worker.
  • Changing what any harness can do for a crewmate: the capability sets (ACP_BACKENDS_MEMBER_DISPATCH, ACP_BACKENDS_MATE_WORKER_KIT) are unchanged.
  • Keeping a live session's native state across a harness change: the chat reloads onto the new harness and its conversation is replayed from the transcript.
  • Showing the control to everyone. It sits behind a Feature Previews switch.

What changed (motivation → approach → change)

A crewmate record gains acp_backend, stored next to its model and worker_model. It uses agent.acp_backend's spelling (kiro, kas, claude, ...), and empty inherits. No second model field: the existing pins are reused and judged against the pinned harness.

Part What it does
Selection members.select_provider_backend takes the pin as crew_backend, one more tier of the one gate: spawn_run(backend=) wins, then the pin, then the member route. Every caller passes it as crewmate_backend_pin(...) or None, and a test fails on one that does not. crewmate_harness.crewmate_for_session finds the crewmate from the session key alone: its DM thread, or its own worker (the worker registry records whose worker it is).
Fails closed A pin this gateway cannot start raises CrewmateHarnessUnavailable (crewmate_harness_unavailable) before any process exists. It never falls back to kiro-cli (#21910). The refusal is a check BEFORE the selection gate (crewmate_backend_pin returns a StartableCrewPin), so the gate itself still never raises (H3); harness-parity.md records the exception as H17. Display reads use crewmate_stored_pin, which never raises. An explicit spawn_run(backend=) override outranks the pin, so the pin is not checked then.
Save PUT /api/agents/{name} takes acp_backend only as "" or a startable harness, and refuses a model / worker_model the pinned harness cannot run, including stored ones when the harness changes.
Read Only the owner-only GET /api/agents/resolved-model returns the pin, plus inherited_harness (what clearing it would run). GET /api/agents leaves it out. GET /api/models?backend= lists one harness's models.
Harness change A save that changes a mate's acp_backend reloads its live DM chat through the existing reload_slot_session (the tab's Reload), clears the slot's own model and effort overrides and rebuilds the provider factory, in one step under the slot and session locks; the record is written only after the old process is down. The transcript and resume pointer stay, so the next message starts on the new harness with the conversation replayed by the existing provider-switch path. While a turn or its sub-agents run, the save is refused with a 409 crewmate_harness_busy that names the mate and says to wait or stop it, and nothing is written. Model and effort picks therefore always reach a session on the saved harness.
Workers A mate's worker run from the sub-agent manager (subagent_manager/run_turn.py) is judged on the mate's pin, then the member route; an unstartable pin refuses the run. crewmate_worker_backend takes the whole config and raises on a config section.
UI An Agent harness row on the mate profile's Settings page, behind Settings > Developer > Feature Previews > Agent harness per mate. The row itself says what it is ("The AI coding tool {name} runs on"), and its value names what Inherited runs on ("Inherited (Kiro Agent)"). The picker's caption explains the rest in plain words, and Inherited there says it follows the crew default. The Model list follows the picked harness; a harness pick puts both model pins back to Inherited. A harness that can't start is named in a notice with a Choose an agent harness button.
Names Every harness the gateway knows gets a readable name (acpBackendName, crewmate_harness.harness_label), so OpenAI Codex, OpenCode, Pi, goose, DeepSeek Harness and GitHub Copilot CLI no longer show as their raw ids, here and on Settings > Agent Harness and first-run setup.

The developer gate is a Feature Previews switch because the Developer Mode switch was removed (docs/request-for-change/rfc-developer-tools-in-settings.md, accepted; #24810). The row is on the profile Settings page, next to the existing Model and Worker model rows, because #25091 folded the crew editor's Advanced section away.

A saved pin rebuilds the provider factory, from the editor or from a config.json edit (session_parts/config_apply.py). The member operating-mode block, the cold-slot readouts, set_model, slash-command checks and kirocrew agent update --model all read the pin through the same helpers. Specs, the user docs and the 12 catalogs plus en-XA are updated.

Where it lives: the pin and its helpers in src/kiro_crew/crewmate_harness.py; the field in config/sections_session.py and config/loader_pipeline.py; the gate tier in members_parts/member_sessions.py (re-exported by members.py), its factory call in config/loader_config.py; the worker backend in mate_worker.py; the dispatch block in context.py and context_assembly/member_gates.py; save and readout in dashboard/agent_admin/crew_update.py and crew_records.py; the CLI check in cli_commands_agent.py; the flag in website/src/utils/previewFlags.ts; the harness names in website/src/api/acpBackend.ts and website/src/i18n/glossary.json; the client calls in website/src/api/client/agents.ts and chatSlotSettings.ts; the copy in website/src/i18n/locales/*.json; docs in src/kiro_crew/docs/configuration.md and the regenerated settings and UI indexes.

Re-port of #11324 by @Zedmor onto current main, scoped to crewmates. Zedmor is credited as co-author on the commit. The call-site guard test and keeping the field out of GET /api/agents come from #16200.

Second commit: main's #25122 and #25127 took subagent_manager/run_turn.py to 2,030 lines, over the 2,000-line cap, so the 9-line worker change above fails check_file_line_cap. The second commit is a behaviour-free move: the body of _run_inner_impl's nested _persist_usage_row goes to subagent_manager/usage_row.py and the nested function calls it, which puts run_turn.py at 1,993 lines. The moved body reads time, asyncio, logger and provider_fallback_active from kiro_crew.subagent at call time, the namespace _run_inner_impl runs on, so existing patches still reach it; test_turn_duration_recorded.py's per-file persist-site count and run_turn's field list move with it.

Backwards compatibility

Compatible: the field is new and defaults to "", which inherits agent.member_acp_backend through the same code path as before; no stored value, endpoint or response field that works today changes meaning.

Removes nothing: the diff adds a field, a request key, two response keys on an owner-only readout, a query parameter and a preview flag; every existing value keeps its answer.

Tests

  • test/test_crewmate_harness.py: gate precedence (override > pin > member route), kiro as a real pin, fail-closed refusal with the crewmate and value named, which session keys take the pin (DM in three spellings, worker, forgotten worker, others), the worker backend, the dispatch block, config round-trip, save validation and no partial write, model judged against the pending and the stored pin, owner readout and its non-owner absence, roster exclusion, hot reload on an agents.*.acp_backend edit, and an AST scan that fails on any select_provider_backend( call without crew_backend.
  • test/test_crewmate_harness.py::TestThePreGateCheck and ::TestSelectionGate::test_the_gate_never_raises_for_a_crew_pin: the refusal happens before the gate, and the gate never raises (H17).
  • test/test_agent_execution_catalog.py: the catalog answers for a mate whose pin cannot start.
  • test/test_crewmate_harness.py::TestAHarnessChangeReloadsTheLiveChat: a harness change reloads an idle live DM chat (process torn down, overrides cleared, transcript and resume kept), an unchanged harness reloads nothing, a running turn or running sub-agents get the 409 with nothing written, and a mate with no live chat is a plain save.
  • test/test_spawn_crewmate_worker.py: a worker run is judged on the member route when unpinned, on the pin when pinned, and is refused on an unstartable pin.
  • website/src/pages/members/CrewProfileSettings.test.tsx: the row is hidden with the preview off; Inherited names what it resolves to; a harness the worker cannot start says so; a pick saves the harness with a model reset and touches no thread; the Model list comes from the pinned harness; a harness that cannot start shows its reason.
  • src/api/acpBackend.test.ts, AgentBackendTab*.test.tsx, KiroPrerequisiteGate.agent*.test.tsx, McpTab.wholeServerDeny.test.tsx: expectations follow the readable harness names.
  • Existing pins updated: the roster allowlist (acp_backend withheld), select_provider_backend call assertions, API_KEY_ORDER in website/src/test/ApiClient.refactor.surface.test.ts, config-baseline.json, settingsRegistry.gen.ts.

Proof: not provable -- other: feature PR; new behaviour has no pre-existing implementation to revert against

Manual verification

Local: the prepare-pr gate floor. On this round, after re-syncing onto main, 3,798 related backend tests and the profile, settings and i18n frontend suites pass. Earlier, the scoped backend suite showed 26 failures that fail the same way on a clean worktree of the base (c6789799df), from this host's environment (symlink-loop and launch-record tests). This round, test_app_agent_launch_copy.py::...[opted-out] fails the same way on a clean worktree of main. The Electron tests need website/electron/node_modules, which this worktree does not have; this PR changes no Electron code. Both are left to CI.

The rendered profile was checked with a capture script against the built dashboard and stubbed API (the screenshots below), asserting the PUT body, the scoped model list, the named unavailable harness and its button, and Inherited's hint while pinned. The Feature Previews card, Settings > Agent Harness and first-run setup were captured from the real components behind stubbed APIs, asserting each harness's readable name.

Screenshots / video

The Feature Previews switch that turns the feature on (light and dark):

Agent harness per mate preview card, light

Agent harness per mate preview card, dark

The Agent harness row on a mate's profile Settings page, with the preview on (light and dark):

Settings page with the Harness row, light

Settings page with the Harness row, dark

The Agent harness picker: Inherited names what it runs on, and a harness the mate's workers cannot start on says so before it is picked.

Harness picker, light

Harness picker, dark

After picking Claude Code: the save puts the model back to Inherited and says when it applies.

Harness saved

Pinned to Claude Code, the Model page lists Claude Code's own models.

Model list scoped to the harness

A pinned harness that can't start, named rather than shown as its id, with the way out inside the notice.

Harness that cannot start

The button opens the picker: Inherited says it follows the crew default, and the pinned harness says only that it can't start here.

Picker opened from the notice

A harness change while the mate is mid-reply is refused, names the mate and says what to do; the pick is put back:

Harness change refused while busy

With the preview off there is no Agent harness row.

Preview off

Every harness by its readable name on Settings > Agent Harness:

Settings > Agent Harness names

And on first-run setup's Use other coding agents:

First-run setup names

Related Issues

Part of #21910
Part of #12451

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@kyleseaman
kyleseaman requested review from a team and KiroCrewCaptain as code owners October 28, 2026 23:43
@kyleseaman
kyleseaman requested a review from bolichen97 October 28, 2026 23:43
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 28, 2026
@github-actions

github-actions Bot commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS (a BLOCK with no Trigger and no Outcome does not block)

Design-level review of 22e49783d3ba3178d5a41c2936466492765e9b8d — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

cannot evaluate: missing screenshot of the new "Agent harness" row, its picker, and the "can't start" notice on the crewmate profile's Settings page; the evidence list has no image or recording for this revision.

Blockers

The new crewmate profile surface has no rendered evidence
Trigger: "An **Agent harness** row on the mate profile's Settings page … A harness that can't start is named in a notice with a **Choose an agent harness** button." This lands in website/src/pages/members/CrewProfileSettings.tsx (new acp_backend row, sub caption, valueLabel "Inherited (…)", a status line after saving), plus a new Feature Previews toggle in FeaturePreviewsSection.tsx.
Outcome: design-evidence.txt lists 0 attachments and 0 committed images, with no transport failures → the row's placement next to Model / Worker model, the caption, the Inherited wording, the notice and the way Model resets after a harness pick cannot be judged → a reshaped profile settings surface would merge unreviewed.
Fix: attach screenshots of this revision with the preview flag on. Show the row at rest, the open picker with per-harness captions, the profile after a harness change (models back to Inherited, saved line shown), and the unstartable-pin notice.
Clears when: the description carries github.com/user-attachments images (or committed files under temp-screenshots/) of those four states, and design-evidence.txt lists them.

Notes

  • Backend shape is sound: the pin is a tier of the existing select_provider_backend gate, refusal happens before the gate (gate still never raises, H3), and a busy session gets a 409 instead of being switched mid-turn.
  • The ## Backwards compatibility section has both a Compatible: line and a Removes nothing: line; keep only Removes nothing, the shape the gate parses.

[DESIGN-REVIEWED] 22e4978

Description read

[DESCRIPTION-READ] 8cc1aff1a7d4cb58bb9b446570e1c8d31ac5a5ede53d86ac60cf2ed5c8986624

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of 22e49783d3ba3178d5a41c2936466492765e9b8d — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Nothing blocks. The pin is opt-in: an empty pin inherits agent.member_acp_backend, the UI control sits behind a Feature Previews switch that is off by default, and every new readout field has a consumer in CrewProfileSettings.tsx.

What this change ships

Inventory (10 items) — 10 justified

Intent: let one crewmate run on a different agent harness, with that harness's models, without moving every other crewmate. ADDITION.

  1. A crewmate record can store its own harness (agents.<name>.acp_backend); empty inherits — justified
  2. A crewmate's chat and its workers start on the pinned harness, ahead of the member route — justified
  3. A pin this gateway cannot start refuses the session instead of falling back to kiro-cli (H17, Let a crewmate (or agent template) choose its own agent harness #21910) — justified
  4. Saving refuses a harness that can't start, and model pins that harness can't run — justified
  5. Changing a mate's harness reloads its live DM chat; a save during a running turn gets 409 crewmate_harness_busy — justified
  6. The owner-only resolved-model readout adds the pin, the inherited harness, the harness list and the error — justified
  7. GET /api/models?backend= lists one harness's models for the picker — justified
  8. An "Agent harness" row on the mate's Settings page, behind a preview flag that is off by default — justified
  9. Codex, OpenCode, Pi, goose, DeepSeek and Copilot show readable names on Settings > Agent Harness and in first-run setup — justified
  10. The usage-row body moves to subagent_manager/usage_row.py to keep run_turn.py under the 2,000-line cap — justified

[FIRST-PRINCIPLES-REVIEWED] 22e4978

Description read

[DESCRIPTION-READ] 3b7acf998b6380a89e018ccaf5552e72e408edce91d13f94f1f8b5de06f2052f

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ✅ PASS (a BLOCK with no Trigger and no Outcome does not block)

UX-level review of 22e49783d3ba3178d5a41c2936466492765e9b8d — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

UX-Verdict: PASS

cannot evaluate: missing screenshots of every new control. No blind read ran because the PR attaches no image, so nobody has checked whether a first-time user understands the Agent harness row, its picker or its notices.

Evidence gaps

  • Agent harness row on the mate profile Settings page, with the sub-label "The AI coding tool {{name}} runs on" and the value "Inherited ({{harness}})". To close: a screenshot of the Settings list with the row showing.
  • Harness picker page: the options, the per-option descriptions ("Follows the crew default (now …)", "…can chat on this harness, but its workers cannot start on it.", "Can't start here.") and the caption profile_harness_hint. To close: a screenshot of the open picker.
  • State after a save: the status line "Saved. {{name}}'s next message starts on this harness." and Model / Worker model reset to Inherited, with the Model list scoped to the picked harness. To close: before and after screenshots.
  • Can't-start state: the profile_harness_unavailable notice with its Choose an agent harness button. To close: a screenshot of the error state.
  • Feature Previews toggle "Agent harness per mate" and its description in Settings > Developer. To close: a screenshot.
  • Renamed harness labels (OpenAI Codex, OpenCode, Pi, goose, DeepSeek Harness, GitHub Copilot CLI) on Settings > Agent Harness and first-run setup. To close: a screenshot of either surface.

Notes

  • profile_harness_hint runs to two sentences and repeats the row's sub-label ("Which AI coding tool runs…"). Cut it down to what happens on a change: "Applies from {{name}}'s next message; Model and Worker model reset to Inherited."

[UX-REVIEWED] 22e4978

Description read

[DESCRIPTION-READ] 6fba66ff0372b7e9e5698fe6fda647d23d0b7844fe031e73184b1c19429f4542

That sha256 names everything this verdict read: the PR description, plus the 3 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 22e49783d3ba3178d5a41c2936466492765e9b8d — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 22e4978

Verdict parsed from the review's SHA-scoped output markers for commit 22e49783d3ba3178d5a41c2936466492765e9b8d.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 22e49783d3ba3178d5a41c2936466492765e9b8d false-positive: <reason, 20 to 500 characters>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 28, 2026
@github-actions

github-actions Bot commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 22e49783d3ba3178d5a41c2936466492765e9b8d and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 22e4978

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 22e49783d3ba3178d5a41c2936466492765e9b8d false-positive: <reason, 20 to 500 characters>

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for scoping the re-port down to crewmates and folding in the call-site guard test. The shape is right: the pin is one more tier in select_provider_backend, the empty value inherits through the same path as today, and the save path refuses a pin or model the gateway cannot run.

Requesting changes at 91d3767 for two blocking items and two smaller ones:

  1. Fail-closed pin vs harness-parity H3. CrewBackendPin.resolve() raises inside the one selection gate, while H3 in docs/system-specs/modules/harness-parity.md still says an unselectable persisted backend never raises and degrades to the Kiro floor with a logged reason. Failing closed is a reasonable product choice here, but it has to be recorded: either add the exception to H3 with a test that pins it, or keep the gate non-raising and refuse earlier, outside it. This PR changes neither the spec nor the gate contract test.
  2. SAST (Semgrep) is red on test/test_crewmate_harness.py:34 and :42: both autouse fixtures patch through the shared monkeypatch, so any test that calls monkeypatch.undo() lifts them. Request _floor_monkeypatch instead.
  3. After a harness pick, the next model pick is pushed to the live slot, which still runs the old harness, so a model only the new harness serves comes back as a 400 even though the record saved it. Skip the live push when the pinned harness differs from the live session's.
  4. UX: "Harness" with the hint "The agent harness ... run on" explains the word with itself. The Settings tab already says "Agent harness"; a hint that says what changes in plain words would help. Please also attach a screenshot of the new Feature Previews card, which the UX lane could not see.

Non-blocking: crewmate_harness._LABELS duplicates names tool_gate already holds for three ids; consider calling label_for for all of them so there is one map.

Checked at this head: diff of the gate, loader, worker, policy, slot_model and model_switch paths; the profile Settings save flow; harness-parity.md and crew-mode.md; the Semgrep log; overlap with the older attempts in this area. Backend and frontend test shards were still pending when I looked.

Comment thread test/test_crewmate_harness.py Resolved
Comment thread website/src/i18n/locales/en.manual.json Resolved
@kyleseaman
kyleseaman force-pushed the feat/crewmate-harness-pin branch from 91d3767 to f86314a Compare October 29, 2026 01:21
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 29, 2026
@kyleseaman

Copy link
Copy Markdown
Contributor Author

Rebased: main moved in docs/system-specs/modules/crew-mode.md and docs/system-specs/modules/learn-cron-dashboard.md since the last head (4bf8b2e -> 8398624), and this push also carries the fixes for the GPT, Opus and UX findings on 91d3767; CI re-running on f86314a.

@kyleseaman

Copy link
Copy Markdown
Contributor Author
  • span=ecaaa9c98064 New-harness model picks still target the old live harness

fixed in f86314a: POST /api/chat/slots/<slot>/model and /reasoning-effort now compare the mate's saved harness with the live session's backend under the existing slot locks. When they differ, the route answers {"ok": true, "next_session": true} before any model/effort validation, slot mutation or live call, so the old harness never judges the pick. The profile page shows the normal "Saved. … next session" status for that reply instead of an error. The effort route had the same defect and is covered too. Regression tests: test/test_dashboard_chat_model_live_switch.py, test/test_chat_slot_reasoning_effort.py, website/src/pages/members/CrewProfileSettings.test.tsx. Repaired by a GPT 6 Astra delegate per review repair routing (served model unverified), verified in the parent.

@kyleseaman

kyleseaman commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor Author
  • span=50c5745326d2 The "Harness" row and its picker are not understood

fixed in f86314a: the row is now "Agent harness", matching Settings > Agent Harness. Its caption says what it is and what it changes in plain words: "Which AI coding tool runs {name}'s chats and its workers, such as Kiro Agent or Claude Code, and so which models {name} can use." The Feature Previews card is "Agent harness per mate" and gives Claude Code as the example.

The evidence gaps and notes are addressed in the same head. The description now shows the Feature Previews card (light and dark), every harness by its readable name on Settings > Agent Harness and on first-run setup, and refreshed profile shots. Inherited always says what it runs on, through a new inherited_harness readout field. The unavailable notice now says "can't start here" and carries a "Choose an agent harness" button that opens the picker.

Follow-up in d67dc12: a model or reasoning-effort pick saved for the next session now gets its own notice ("Saved. {name}'s next session uses this model." / "… uses this reasoning effort."), new keys profile_model_next_session and profile_effort_next_session in all catalogs, instead of reusing the harness wording.

These copy and UI changes were made by the author directly, not by a model-pinned delegate.

@kyleseaman
kyleseaman force-pushed the feat/crewmate-harness-pin branch from f86314a to bdb5df5 Compare October 29, 2026 01:24
@kyleseaman

Copy link
Copy Markdown
Contributor Author
  • span=e0a2334fbbbe Unstartable crewmate harness pin raises inside the selection gate (harness-parity H3)

fixed in f86314a: took the review's fail-closed branch, since failing closed is the owner's stated requirement for this feature. The refusal now happens before the gate: crewmate_backend_pin checks the pin and raises CrewmateHarnessUnavailable for one this gateway cannot start, and otherwise returns a StartableCrewPin. select_provider_backend takes only that type, returns its backend and never raises, so H3 holds for the gate itself. The exception is recorded as a new Group A row H17 in docs/system-specs/modules/harness-parity.md, and H3 points to it. It is pinned by test_crewmate_harness.py::TestThePreGateCheck (incl. a harness that stopped being selectable after save, and the provider factory refusing with the gate never called) and ::TestSelectionGate::test_the_gate_never_raises_for_a_crew_pin; the caller scan now also fails on an unchecked crew_backend value. Display reads use the non-raising crewmate_stored_pin (the agent catalog, with a regression test). Repaired by an Opus 5.5 delegate per review repair routing (served model unverified), verified in the parent.

@kyleseaman

Copy link
Copy Markdown
Contributor Author

@bolichen97 thanks for the review. All four items are addressed at bdb5df5 (replies on each thread). On the non-blocking label map: tool_gate.label_for already names Kiro CLI, so crewmate_harness._LABELS now holds only Kiro Agent and Claude Code, the two ids label_for answers with their raw id; everything else comes from that one map. Folding those two into tool_gate too would change its operator-facing labels for other callers, so I left that out of this PR.

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the quick turnaround. At bdb5df5 the four points from my last review are addressed: H3 now defers to the new H17 row with tests that pin the pre-gate refusal and a gate that never raises, the autouse fixtures use _floor_monkeypatch (SAST is green), the profile picker no longer gets a 400 after a harness switch, and the copy reads "Agent harness" with a plain hint. The design is good to go from my side.

Still requesting changes for three items at this head:

  1. crewmate_worker_backend changed from taking AgentConfig to taking the whole config plus the crewmate, but subagent_manager/run_turn.py:317-321 still passes .agent and no crewmate. getattr(AgentConfig, "agent") is None, so member_acp_backend reads as empty and the crewmate's pin is never consulted on that path. Before this PR the same call returned the member backend. Please pass the loaded config and crew_for_member_id(...), and treat CrewmateHarnessUnavailable as a refusal there. A test on that call site would catch the signature drift.
  2. The next_session shortcut in slot_model.py (and the same one in slot_effort.py) answers 200 for any model or effort pick on a crewmate session whose configured harness differs from the live one. Only the profile page saves the record first; the composer and keyboard pickers post to the same route, read nothing from next_session, and show a pick that was stored nowhere. The shortcut also returns before _model_rejected_reason, so an invalid id gets 200. Please scope it to the profile save (for example an explicit request flag), refuse other callers with a 409 naming the pending harness change, and validate against the configured harness in both cases.
  3. Backend shards 2 and 5 (Linux) and 5 (Windows) are red. test_backends_vocabulary_family.py::test_no_product_module_but_the_facade_imports_an_owner fails because crewmate_harness.py imports ACP_BACKENDS_MATE_WORKER_KIT from agent_sdk.backends_members instead of the agent_sdk.backends facade. Shard 2 also fails test_slot_reset_interleaving_seam.py::test_switch_blocks_until_reloads_serialized_teardown_completes ("switch never completed after reload released the locks"); this PR adds work to the same api_chat_slot_model path, so please check whether it is related before rerunning.

Checked at this head: range-diff against 91d3767, the H3 and H17 rows, the test fixtures, slot_model.py, slot_effort.py, mate_worker.py, every crewmate_worker_backend caller, the profile settings save flow, and the failing job logs.

Comment thread src/kiro_crew/mate_worker.py Resolved
Comment thread src/kiro_crew/crewmate_harness.py Resolved
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 29, 2026
@kyleseaman
kyleseaman force-pushed the feat/crewmate-harness-pin branch from bdb5df5 to d67dc12 Compare October 29, 2026 02:19
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 29, 2026
@kyleseaman

Copy link
Copy Markdown
Contributor Author
  • span=aa8f4d029522 Ordinary chat selections report success without retaining the pick

fixed in d67dc12: next_session is now answered only to a caller that says the record already holds the pick: the profile page saves to the mate's record, then posts with profile_saved: true. With a saved harness that differs from the live session's, every other caller (the composer, sessionPicks, the keyboard shortcuts, the effort dropdown, the crew window, Mochi) gets a 409 crewmate_harness_pending naming the pending agent harness change, so nothing reports success without saving. Flagged picks are still validated against the saved harness (invalid model 400, effort outside the vocabulary 400) before success. Covered for DM and worker sessions in test_dashboard_chat_model_live_switch.py and test_chat_slot_reasoning_effort.py, and CrewProfileSettings.test.tsx checks the flag. Repaired by a GPT 6 Astra delegate per review repair routing (served model unverified); the parent changed the 409 wording to say "mate" and "agent harness" and removed an assert.

@kyleseaman

Copy link
Copy Markdown
Contributor Author
  • span=b5a74841df93 Crewmate worker runs from the subagent manager are judged on the wrong harness

fixed in d67dc12: subagent_manager/run_turn.py now loads the whole config and calls crewmate_worker_backend(cfg, crew_for_member_id(cfg, member_id)), so an unpinned run is judged on agent.member_acp_backend (kas by default) and a pinned one on its pin; CrewmateHarnessUnavailable becomes the run's refusal, led by its code, with no fallback. crewmate_worker_backend is typed to take the whole config and raises TypeError on a config section, so a stale caller fails loudly. A sweep of every caller of the helpers whose signature this PR changed found no other. Regression tests in test_spawn_crewmate_worker.py fail with the old call. Repaired by an Opus 5.5 delegate per review repair routing (served model unverified), verified in the parent.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 29, 2026
kyleseaman pushed a commit that referenced this pull request Oct 29, 2026
…tion (H17)

A mate's own agents.<name>.acp_backend pin that cannot start refuses that
mate's sessions before the selection gate instead of degrading to kiro.
Add it to the harness-parity review rule as the one H1/H3 exception,
with what stays flagged. The H17 spec row lands with the code in #25485.
@kyleseaman

kyleseaman commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor Author

Rebased onto main a03e1621fece to clear another conflict in the generated src/kiro_crew/docs/ui-index.generated.json, regenerated with npm run gen:ui. No code change. New head 75d017de86644a9f8f6a8d72ffcc88553a4fa225. Local gate (generated files clean, i18n, tsc, members/settings vitest, targeted pytest, harness parity, docs lint, comment history, line cap, mypy, flake8, black) passes.

#25798 is now AUTOSDE.yaml only; the H17 row in harness-parity.md stays in this PR with the code it cites.

@kyleseaman
kyleseaman force-pushed the feat/crewmate-harness-pin branch from 95115e5 to 75d017d Compare October 29, 2026 16:59
@github-actions github-actions Bot removed the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Oct 29, 2026

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed at 75d017d. I read the full range-diff from d3b7302.

Fixed since my last review: unrelated edits to a crewmate whose stored harness cannot start. judges_a_model_edit now skips an inherit value and a value equal to the stored one unless the harness moves, and it is applied in the pre-lock probe and in the locked check for both model and worker_model. The new tests cover a re-sent model, worker_model and both, and also check that a harness move still judges unchanged pins. That clears the red item from my previous review. I also checked that the editor sends acp_backend only when the harness row itself is picked, so a re-sent unstartable pin does not hit harness_pin_rejected on an ordinary save.

Still blocking: the Opus finding on this head is valid. CrewBackendPin.resolve maps a stored kiro pin with backend_from_stored and returns ACP_BACKEND_KIRO as is. The Kiro CLI choice that main now moves onto KAS once v3 is confirmed (kiro_cli_pick_backend, applied in _read_backend_choice) is never applied to the crewmate pin. So a crewmate pinned to Kiro CLI keeps the v2 launch while the same spelling in agent.acp_backend runs on KAS, and agent.member_acp_backend = "kiro" is always remapped to KAS. Details inline.

Still open, not a code defect: the base harness-parity rule on main has no H17 exception yet. The companion rule PR carries it and should land first.

This replaces my earlier change requests. Once the pin follows the Kiro CLI mapping and the AI lanes and readiness pass on the new head, I expect to approve.

Comment thread src/kiro_crew/crewmate_harness.py Resolved
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 29, 2026
@kyleseaman
kyleseaman force-pushed the feat/crewmate-harness-pin branch from 75d017d to 8f2bdde Compare October 29, 2026 18:08
@kyleseaman

Copy link
Copy Markdown
Contributor Author
  • span=5db5d2e29432 A crewmate pinned to kiro always gets the v2 Kiro CLI launch

fixed in 8f2bdde: a stored kiro pin now reads the way a stored agent.acp_backend does in _read_backend_choice. CrewBackendPin carries agent.kas_bundle from the same config, and resolve() returns kiro_cli_pick_backend() for a selectable kiro pin off the bundle route, so it runs on KAS once the host's kiro-cli is confirmed to offer v3 and while KAS is selectable, on the DM chat and the worker path alike. The selection gate does not change, and an unselectable pin still fails closed (H17). The agent catalog reports the harness the pin runs on. Pinned by test_crewmate_harness.py (test_confirmed_v3_runs_the_dm_and_the_worker_on_kas, test_an_unconfirmed_kiro_cli_keeps_the_kiro_cli_launch, test_the_kas_bundle_route_keeps_the_kiro_cli_launch, test_kas_not_selectable_keeps_the_kiro_cli_launch, test_a_kiro_pin_on_v3_dispatches_like_an_inherited_kiro, test_a_kiro_pin_on_v3_reads_as_stored_and_runs_on_kas); the H17 row, crew-mode.md and configuration.md say so. Repaired by an Opus 5.5 delegate per review repair routing (served model unverified), verified in the parent.

Also in this push: rebased onto main 043dd9fd7ff6. Main's #25598 added a select_provider_backend caller in dashboard/chat_api/slot_settings.py (the auto-compact hint); the PR's call-site guard (TestEveryCallerPassesThePin) caught it, and it now passes crew_backend=crewmate_backend_pin(cfg, key).

@kyleseaman

Copy link
Copy Markdown
Contributor Author
  • span=0f24a0f7bb7f Dormant chats retain model overrides cleared by a harness save

fixed in 1acd024: a save that changes a mate's acp_backend now also clears the persisted model and reasoning_effort of the mate's bound DM chat when that chat is not open, right after the record is written. It uses the history's guarded metadata writer (update_metadata_if(..., require_existing=True)), so the transcript and resume pointer stay, the tab is not reopened, and a deleted session is never recreated. This is the same clearing the open-chat reload branch already does, and it also runs when another DM slot is live. An unchanged harness touches nothing. Pinned by new tests in test_crewmate_harness.py::TestAHarnessChangeReloadsTheLiveChat::test_a_dormant_dm_reopens_without_old_harness_picks and ::test_an_unchanged_harness_preserves_dormant_picks (a closed DM with a persisted model and effort restores neither after a harness change, for both memory-store spellings; an unchanged harness keeps them). crew-mode.md (Harness change) says so. Repaired by a GPT 6 delegate per review repair routing (served model unverified), verified in the parent. Also rebased onto main ae1bb2dd1bf1.

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-review at 1acd024cd6ceb0690247109db42897c55f3341bb (range-diff from 75d017de86644a9f8f6a8d72ffcc88553a4fa225).

The blocking finding from my last review is fixed. CrewBackendPin.resolve now sends a stored kiro pin through kiro_cli_pick_backend() when agent.kas_bundle is off, the same way _read_backend_choice reads a stored agent.acp_backend. Both construction sites (crewmate_pin_for and pinned_entitlement_backend) pass kas_bundle, the display path in agent_catalog._session_backend reads the resolved harness, and the new TestAKiroPinOnAV3KiroCli tests plus the H17 row cover both routes. The other hunks in the range-diff (dormant DM model and effort clearing on a harness save, the autocompact backend lookup passing crew_backend, docs) read clean. No new blocking findings from me.

What still blocks approval at this head:

  • The branch conflicts with main again, in website/src/i18n/en.context.json and the locale catalogs. Please rebase.
  • CI is still running.
  • The Opus lane blocks on the H17 exception being absent from the base harness-parity rule. That rule change is #25798, and it needs to land first.

Non-blocking, from the Design lane: the crewmate worker registry lives only in process memory, so a worker resumed after a gateway restart follows the member route rather than the pin. Worth a sentence in H17 or a follow-up.

I will re-check once the rebased head is green.

@kyleseaman

Copy link
Copy Markdown
Contributor Author

Rebased onto main 09e4e0bfe296 to clear a conflict in the 12 i18n catalogs (website/src/i18n/locales/*.json and en.context.json): main's #25091 appended keys at the same place as this PR's Agent harness strings. Kept both sides, and checked that each catalog holds exactly main's keys plus the keys this PR adds; i18n:pseudo, gen:ui and gen:settings regenerate with no change.

#25091 also grew dashboard/handlers/agents.py past the 2,000-line cap with this PR on top, so this PR's own changes there are shorter: crewmate_harness.models_route now takes the query mapping, and two docstrings are condensed. No behaviour change. New head 2bdd6e561454c81b51e211533c46a0b92a5ce8a4.

Local gate: generated files clean, i18n (vitest + i18n:check), tsc, members/settings vitest, targeted pytest including the agents composition and error-code contracts, harness parity, docs lint, comment history, line cap, mypy, flake8 and black all pass.

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-review at 2bdd6e5 (range-diff against 1acd024).

The new head changes look correct. models_route now takes the request query mapping, so only a present backend key asks for a harness; the handler passes request.query and the tests cover an empty value, an unselectable harness and no query. The other code edits are comment and naming tidy-ups, and the locale and context JSON edits are the conflict resolution plus the same three harness strings. No new findings.

This branch conflicts with main again: the generated ui-index.generated.json under the docs package changed on main after your rebase (the Sprites and boxd session environments change). Please rebase onto current main and regenerate that index file. Once the branch is clean and the checks are green, the remaining gate is the harness parity rule change in the companion spec PR, which needs to land first.

@kyleseaman

Copy link
Copy Markdown
Contributor Author

Rebased onto main e598df5478f8 to clear another conflict in the generated src/kiro_crew/docs/ui-index.generated.json, regenerated with npm run gen:ui. New head 4e49af70796413fce81267fc8be04f49a19a21ce.

The PR now has two commits:

Local gate: generated files clean, i18n, tsc, members/settings vitest, targeted pytest (8,064 subagent and related tests; the one failure, test_subagent_prompt_hooks.py::test_hook_time_is_not_charged_to_the_first_prompt_watchdog, also fails on main e598df5478f8), harness parity, docs lint, comment history, line cap, mypy, flake8 and black.

@kyleseaman

Copy link
Copy Markdown
Contributor Author
  • span=ea58ae1e9d46 A harness save with two or more live DM chats can reload the first, post a false notice, then answer 409

fixed in 91444b8: a harness change for a mate with more than one live DM chat is now refused with 409 crewmate_harness_busy before any chat is torn down ("{name} has more than one chat open. Close the mate's other chat, then change the agent harness."), and nothing is written. So the nested reload never starts. The approval-mode compare-and-set runs before the reload as well (memory_stores.check_member_approval_mode, the same check persist_member_config makes, under the config lock and without writing), so an approval conflict refuses before any teardown; the in-hook write still checks again. The single-chat and no-chat paths are unchanged. Pinned by new tests in test_crewmate_harness.py (::test_a_second_live_chat_refuses_before_any_teardown, ::test_an_approval_conflict_refuses_before_the_chat_is_torn_down; the single-chat ::test_an_idle_chat_is_reloaded_onto_the_new_harness is unchanged). crew-mode.md (Harness change) says so. Repaired by an Opus 5.5 delegate per review repair routing (served model unverified), verified in the parent. Also rebased onto main b5dc1567f1c6.

@kyleseaman

Copy link
Copy Markdown
Contributor Author

Rebased onto main b5dc1567f1c6. New head f4d84c67c2f2bc5aa67a368f0f4ab47cfd1b9c9c; the PR is still two commits:

  • 91444b85096bd8cce9d6d36fac25b0e4200127f0, the feature, now with the multi-chat refusal and the approval check before teardown (the disposition above).
  • f4d84c67c2f2bc5aa67a368f0f4ab47cfd1b9c9c, the behaviour-free usage_row.py move. It now also lists usage_row.py in test/metrics/test_usage_turns.py's scan of the run_* family, which the move had hidden from that test (the Backend Tests shard 3 red on 4e49af70).

Frontend Tests (1) on 4e49af70 failed in src/test/kiroUsageRefetchInterval.test.ts; that test fails on main too and #25944 fixes it.

Local gate: generated files clean, i18n:check, tsc, members/settings vitest, targeted pytest, harness parity, docs lint, comment history, line cap, mypy, flake8 and black pass; the i18n vitest run passed except navLabels.test.tsx, which timed out under load and passes alone.

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-review at f4d84c6 (range-diff against 2bdd6e5 and 4e49af7).

The new commits look correct and I have no new findings.

  • The harness save now refuses with 409 crewmate_harness_busy when the mate has more than one live DM chat, before any teardown, so the nested reload that could post a false notice and then answer 409 cannot start.
  • check_member_approval_mode runs the same compare-and-set as persist_member_config under the config lock, and its mutate returns None, so update_config_locked skips the write. An approval conflict is now refused before the chat is reloaded; the in-hook write still checks again.
  • Moving the usage row into usage_row.py broke test_usage_turns.py::TestWriteSiteStamping::test_subagent_completion_stamps_the_dispatching_app at 4e49af7 (the source pin only read run.py and run_*.py). The test now reads usage_row.py as well, and it passes locally at this head, together with test_crewmate_harness.py, test_spawn_crewmate_worker.py, test_mate_worker_dispatch.py and test_turn_duration_recorded.py.

CI note: at 4e49af7, three backend failures (test_dir_fd_create_race_ratchet.py, test_security_gate_liveness.py line budget, test_subagent_prompt_hooks.py watchdog) also fail on the base commit. They still fail locally at this head, whose base predates the main fix for them. Rebasing onto current main should clear them; the branch merges cleanly with it.

Merge order is unchanged: this PR should land after the companion rule change for the H17 exception.

Kyle Seaman and others added 2 commits October 29, 2026 16:28
A crewmate can name the agent harness its chat and the worker sessions it
opens run on (agents.<name>.acp_backend), beside the model it already pins.
Unset inherits agent.member_acp_backend exactly as before. The pin is a tier
of the one selection gate (members.select_provider_backend), below an explicit
spawn_run(backend=...) and above the member route, and every caller passes it.
A pinned harness this gateway cannot start refuses the session with
crewmate_harness_unavailable instead of falling back to kiro-cli. The model and
worker_model pins are judged against the pinned harness at save time.

The dashboard shows a Harness row on the crewmate's profile Settings page,
behind a Settings > Developer > Feature Previews switch, and scopes the Model
list to the picked harness. The pin is read only through the owner-only
GET /api/agents/resolved-model; GET /api/agents leaves it out.

Re-port of #11324 onto current main, scoped to crewmates.

Co-authored-by: Akim Akimov <9255+Zedmor@users.noreply.github.com>
…stay under the line cap

main took subagent_manager/run_turn.py to 2,030 lines, over the 2,000-line
cap, so any change to it fails check_file_line_cap. Move the body of
_run_inner_impl's nested _persist_usage_row into a sibling module,
subagent_manager/usage_row.py, and call it from the nested function.

No behaviour change: the moved body reads time, asyncio, logger and
provider_fallback_active from kiro_crew.subagent at call time, the
namespace _run_inner_impl runs on, so existing patches still reach it.

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving at head 22e4978.

Gates checked at this head: label readiness: passed; every check green or skipped, none red or pending; Design, First Principles, UX, Opus and GPT review lanes all PASS; Sensitive Change Review green; mergeable with no conflict; base is main; no other human change request.

Every hunk of the diff (87 files) was read across our review rounds. This head is a rebase of f4d84c6: range-diff shows both commits unchanged and the whole-PR patch-id is identical, so nothing new needed reading. Areas covered:

  • Crewmate harness pin and its pre-gate check (new crewmate_harness.py, policy.py backend selection, config apply rebuild on an acp_backend edit, member dispatch gates).
  • Crew editor save path in crew_update.py and crew_records.py, including the approval-mode compare-and-set run before the chat reload, which only adds a refusal and never widens one.
  • DM chat and crewmate worker refusal when a pinned harness cannot start (fails closed, no fallback to another harness), plus slot model, effort and session create paths.
  • Harness-parity spec H17 row and crew-mode docs, matching the H17 rule now on main.
  • The usage_row move out of run_turn.py into its own module (refactor, no behavior change).
  • Tests (new harness, dispatch, live model switch and spawn tests), locales and generated settings and UI index files, and the crew profile settings page.

Security checklist answered at this head: no change to auth or owner checks, no new unvalidated external input (the pin is checked against the selectable harness set), no trust boundary or sandbox change, approval mode handling only gets stricter, no secret handling.

This approval replaces our earlier request for changes. The findings from that round (the multi-chat harness save and the missing H17 rule) are fixed in this head and in the companion spec change that has merged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants