Skip to content

docs(rfc): crewmate pin that cannot run a full crewmate runs on KAS with an error; model follows harness - #30116

Merged
kyleseaman merged 1 commit into
mainfrom
docs/rfc-bot-harness-amend
Nov 5, 2026
Merged

kyleseaman merged 1 commit into
mainfrom
docs/rfc-bot-harness-amend

Conversation

@iamwhatever

@iamwhatever iamwhatever commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Record the owner's 2026-11-05 changes to the crewmate-harness RFC, so the implementation PRs trace to the rules the owner now wants.

The draft RFC docs/request-for-change/rfc-crewmate-harness-follows-settings.md (#29442) says a crewmate pinned to a harness this gateway cannot start refuses (H17), and it says nothing about which model a crewmate uses when it lands on a different harness. On 2026-11-05 the owner changed both rules.

Why it matters

Today a hand-edited pin to an unrunnable harness stops that crewmate entirely, and a crewmate that moves to another harness can start with a model that harness cannot serve. The implementation PRs need the new rules written down before the First Principles and Design lanes can check code against them.

Not a goal

  • No code changes.
  • No change to the RFC's status field; acceptance is the owner's call.
  • No change to the H17 refusal for a pin this gateway cannot start.

What changed (motivation → approach → change)

Owner decisions → write them into the RFC as O6 to O8, and update the H17 rule text in the same docs PR (the #25798 precedent: rule changes go in docs PRs, not code PRs) → three files.

  • O6: the picker lists only harnesses a crewmate can run on: KAS, Claude Code, Codex (5.7).
  • O7, new section 5.8: a pin this gateway cannot start still refuses (H17). A pin that starts but cannot run a full crewmate (no crewmate chat or no worker kit: goose, opencode, strands, pi, deepseek, copilot; new named set ACP_BACKENDS_CREWMATE_PIN_TO_KAS) runs on KAS, with an error in the editor, the log and the dashboard bell. That KAS still crosses the selection gate. copilot is in the set because it meets the definition (selectable, in none of the crewmate sets). Codex is not: on main it has the worker kit, so a Codex pin runs on Codex.
  • H17 in docs/system-specs/modules/harness-parity.md and the H17 Allowed entry of rule harness-parity in AUTOSDE.yaml now say the same, with what reviewers still flag.
  • O8, new section 5.9: a crewmate's model follows its effective harness. Its own model is used when valid there; otherwise the Settings model on the Settings harness, else that harness's default.
  • Section 7 adds Phase 6 (pin fallback) and Phase 7 (model follows harness), each with named tests.

The Not a goal bullet that excluded the H17 edit was replaced, because the owner then asked for the H17 change in this PR.

The RFC cites symbols, not line numbers; each new symbol was checked at main.

Security checklist (AUTOSDE.yaml edit)

  • Gate touched: the H17 entry under Allowed in the blocking rule harness-parity. No blocking: flag, file pattern or other rule changes.
  • Loosened: one case only. A startable pin to goose, opencode, strands, pi, deepseek or copilot is now allowed to run on KAS instead of on the pinned harness. Before, that pin ran on the pinned harness, so no refusal is removed.
  • Kept: the fail-closed refusal for a pin this gateway cannot start, the D6 enterprise floor, policy denials and the per-session KAS refusal (the KAS still crosses the gate).
  • Added FLAG clauses: running on the pinned harness, a silent fallback, handing the gate anything but KAS, applying the fallback to an unstartable pin or to any session other than that crewmate's DM or worker.
  • No credential, sandbox, network or auth surface changes; no code changes.

Backwards compatibility

Compatible: documentation and review-rule text only; no code, config or schema changes.

Tests

Proof: not provable -- docs or config only

python3 scripts/docs_lint.py reports "All documentation checks passed"; AUTOSDE.yaml parses; test/test_harness_parity.py and the GPT/Opus review-workflow tests pass except two that fail the same way on main in this environment.

Manual verification

Both local reviewers ran on the CI review contracts before the push. GPT: no findings. Opus: no Critical/High; one Medium (the rule text required the error to name KAS, while 5.8 lands on the Kiro CLI floor when policy denies kas) and two Low (pi/deepseek pin vs inherit reasoning; a stale "covers deepseek and pi" line), all fixed. CI Opus then found copilot missing from the set (verified: ACP_BACKEND_COPILOT is in BASELINE_SELECTABLE_BACKENDS and in none of ACP_BACKENDS_MEMBER_CAPABILITIES, ACP_BACKENDS_MEMBER_DISPATCH, ACP_BACKENDS_MATE_WORKER_KIT); fixed, with a two-way set test named in Phase 6.

Related Issues

Refs #21910

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributions are accepted under the project's Apache-2.0 license.

@iamwhatever
iamwhatever requested review from a team and KiroCrewCaptain as code owners November 5, 2026 02:06
@github-actions github-actions Bot added the readiness: action required A blocking check or review needs attention label Nov 5, 2026
@github-actions

github-actions Bot commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ⏭️ skipped

Revision 7bbe80774c9e4c15f65c07d6bbe4058e9ab68a2e ships no reviewable capability (docs, tests or generated files only), so there is nothing to inventory. Does not block merge.

@github-actions

github-actions Bot commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 7bbe80774c9e4c15f65c07d6bbe4058e9ab68a2e touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 7bbe80774c9e4c15f65c07d6bbe4058e9ab68a2e — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: this is a docs-only change that writes down decisions the owner made, the H17 rule is updated in harness-parity.md and AUTOSDE.yaml together, and the unstartable-pin refusal stays.

Notes

  • ACP_BACKENDS_CREWMATE_PIN_TO_KAS is a hand-written list, but its definition depends on three other sets. When Phase 6 lands, the two-way set test needs to fail if anyone adds a new selectable harness without deciding whether it belongs in the list. Otherwise that harness ends up as a pinned crewmate that can't run, with no error shown.

[DESIGN-REVIEWED] 7bbe807

Description read

[DESCRIPTION-READ] 587de6e8bead3a57739bf1ff26acddb8f3205dfbc685b2bec394866699747631

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 7bbe80774c9e4c15f65c07d6bbe4058e9ab68a2e and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 7bbe807

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 7bbe80774c9e4c15f65c07d6bbe4058e9ab68a2e false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 7bbe80774c9e4c15f65c07d6bbe4058e9ab68a2e — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 7bbe807

Verdict parsed from the review's SHA-scoped output markers for commit 7bbe80774c9e4c15f65c07d6bbe4058e9ab68a2e.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 7bbe80774c9e4c15f65c07d6bbe4058e9ab68a2e false-positive: <reason, 20 to 500 characters>

@bolichen97

Copy link
Copy Markdown
Contributor

Thanks for this PR. The only real reds on this head are Backend Tests (3.12, 7) and (Windows) (7), which come from a Slack code-channel test regression on main, not from your change. That is now fixed on main. Could you update the branch onto the latest main? We will review it again once CI is green.

@iamwhatever
iamwhatever force-pushed the docs/rfc-bot-harness-amend branch from 60958d0 to 1f62274 Compare November 5, 2026 04:38
@iamwhatever iamwhatever changed the title docs(rfc): crewmate pin to an unrunnable harness runs on KAS with an error; model follows harness docs(rfc): crewmate pin that cannot run a full crewmate runs on KAS with an error; model follows harness Nov 5, 2026
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Nov 5, 2026

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment on 1f62274. This is an RFC, so the sign-off is a maintainer decision; here is a short brief plus one point to fix first.

Design brief

What it does: a crewmate follows the harness the user picks in settings. A harness that cannot run a full crewmate (no crewmate chat or no worker kit) falls back to KAS. That fallback is keyed on a new explicit set, ACP_BACKENDS_CREWMATE_PIN_TO_KAS.

Pros: one rule for which harness a crewmate runs on. An unsupported harness falls back to a working one instead of a half-working crewmate.

Cons: the fallback is only as good as the set. A harness missing from it runs silently on the pinned harness.

Long-term tech debt: low, provided the set is checked against the selectable list in both directions.

One point before sign-off

The Opus lane is right, and I checked it on main. ACP_BACKEND_COPILOT is in BASELINE_SELECTABLE_BACKENDS (agent_sdk/backends.py), but it is in none of ACP_BACKENDS_MEMBER_CAPABILITIES, ACP_BACKENDS_MEMBER_DISPATCH or ACP_BACKENDS_MATE_WORKER_KIT (agent_sdk/backends_members.py). By the RFC's own definition at rfc-crewmate-harness-follows-settings.md:212 it cannot run a full crewmate, yet the 5.8 set lists only goose, opencode, strands, pi and deepseek. Please add copilot to the set (and the matching harness-parity.md and AUTOSDE.yaml text), or record why it is left out. Please also make the planned set test check the other direction: every selectable harness outside the crewmate sets must be in ACP_BACKENDS_CREWMATE_PIN_TO_KAS.

@bolichen97

Copy link
Copy Markdown
Contributor

Maintainer direction on this RFC: signed, with two conditions before merge.

  1. Add copilot to the crewmate set or to ACP_BACKENDS_CREWMATE_PIN_TO_KAS, with the matching harness-parity.md and AUTOSDE.yaml text.
  2. Add a reverse test that fails when a selectable harness is in neither the crewmate sets nor ACP_BACKENDS_CREWMATE_PIN_TO_KAS.

Once both are in, this is good to merge from the design side.

…ith an error; model follows harness

Amends rfc-crewmate-harness-follows-settings with the owner's 2026-11-05
decisions O6-O8 and updates H17 in harness-parity.md and AUTOSDE.yaml to
match: an unstartable pin still refuses; a startable pin to goose,
opencode, strands, pi or deepseek runs on KAS with an error log and bell.
Status field unchanged.

Refs #21910
@iamwhatever
iamwhatever force-pushed the docs/rfc-bot-harness-amend branch from 1f62274 to 7bbe807 Compare November 5, 2026 05:29
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Nov 5, 2026
@bolichen97

Copy link
Copy Markdown
Contributor

Both sign-off conditions are met at 7bbe807:

  1. copilot is now in ACP_BACKENDS_CREWMATE_PIN_TO_KAS alongside goose, opencode, strands, pi and deepseek (O6, O7, the 5.8 table and Phase 6's test_unrunnable_pin_runs_on_kas cases).
  2. Phase 6 adds the reverse test test_harness_parity.py::test_every_selectable_harness_without_chat_or_worker_kit_is_in_pin_to_kas.

Thanks. The only red entries on this head are cancelled older duplicate runs.

@kyleseaman
kyleseaman merged commit b552e58 into main Nov 5, 2026
100 of 108 checks passed
@kyleseaman
kyleseaman deleted the docs/rfc-bot-harness-amend branch November 5, 2026 16:22
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Nov 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants