Repository navigation
ci(review): move every AI review lane to Opus 5.5 and GPT 6.1 Sol - #16238
Conversation
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS The GPT adjudicator's Claude Code 2.1.287 pin and its Sonnet 5.5 fallback have never run. Check the first adjudicator run on [DESIGN-REVIEWED] a20d9e1 Description read[DESCRIPTION-READ] 81eb15a432b5b8de46cbabc822f507189fd6ae20b59565495388b7fba55d7689 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5) — 🟡 CONCERNSPremise-level review of First-Principles-Verdict: CONCERNS The prepare-pr repair ladder drops Fable 5.1 and Fable 5, but the Goal only covers CI. This ladder is a local skill preference, and the skill says CI models do not set it. Not justified as shipped
What this change shipsInventory (10 items) — 9 justifiedIntent: the maintainer wants every CI review lane on newer, cheaper models with Fable gone from CI. This is an ADDITION (a model upgrade).
[FIRST-PRINCIPLES-REVIEWED] a20d9e1 Description read[DESCRIPTION-READ] a0230c642b670db10c8325d4d59f3b885a4934fb9c843aa7aa784b9a1fc06917 That sha256 names the PR description this verdict read. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
Security Scope Review -- 🟡 CONCERNSWhich legitimate operations does Verdict basis: the reviewer's own judgement; no confirmed regression. Security scope review -- adjudicated candidates
No candidate flipped from allowed to refused. Scope-Verdict: CONCERNS TIGHTENING: The only control this diff makes stricter is the CI version floor on the adjudicator's Claude Code CLI. It refuses to run the Opus 5.5 adjudication stage on a Claude Code CLI too old for the new primary and fallback models. The hunk in ADJUDICATED: 12 candidates, 0 confirmed newly-refused (no verdicts had come back when this review was written; they are classifier rows, not claimed results) PLATFORM COVERAGE: The diff adds no path, separator, interpreter or env-var assumption to the deny composite. The candidates still cover the operations around this change in every spelling:
The CI-side floor runs only on UNADJUDICATED: The floor check is a CI version gate in the "adjudicator floor" step of
Model ids and check names also moved ( [SCOPE-REVIEWED] a20d9e1 |
dbe8181 to
839de29
Compare
|
|
|
Frontend Tests (1)/(2) are red on this head for reasons this PR does not touch (it changes no
I'll rebase onto main once one of those lands and let CI re-run. |
|
|
|
Rebased onto main to pick up the frontend test fix (#16227); CI re-running. |
839de29 to
6ab3444
Compare
6ab3444 to
7141b0c
Compare
|
|
|
7141b0c to
a20d9e1
Compare
|
Rebased onto main (9a59174). Head 7141b0c lost its Security Scope check to the per-head floor: two earlier runs failed on the shared GitHub App rate limit, and the third run (CONCERNS, 0 newly refused) could not clear that. A new head re-measures. First Principles items on 7141b0c were answered in 5964438763 and 5964438901. |
|
buluoray
left a comment
There was a problem hiding this comment.
Summary
The model swap itself is correct and consistent. Every --model/--fallback-model/model =/MODEL_IDS default under .github/ now names claude-opus-5-5, claude-sonnet-5-5 or openai.gpt-6.1-sol. All 16 claude-code-action uses are pinned to 97c53473…, which is the v1.0.239 tag. The readiness triggers, the fork check-run names, the override table and the docs all use the new lane names. No secret, permission, role or region changed. Auto-merge is on, so I am applying the stricter bar: one stale workflow header still names the old model, and that is a one-line fix. One inaccurate PR-body sentence is listed as a follow-up.
Blocking
.github/workflows/codex-review.yml:3still says# Second AI reviewer using OpenAI GPT-5.6 Sol via Amazon Bedrock (Mantle).The workflow is nowname: GPT 6.1 Review(line 1) and runsmodel = "openai.gpt-6.1-sol"(line 235). This PR edited lines 1 and 9 of the same header block but missed this one. It is the only old-model description left under.github/. Line 1023 (#9216 … Opus 4.8's 2.1.255 floor) andrelease.yml:304are historical incident notes, so they are fine.
This is the only blocker, and with auto-merge on it would otherwise ship as-is.
What I verified
- Old ids. I grepped the head tree for old ids (
claude-opus-5not followed by-5,opus-4-8,Opus 4.8,fable-5,Fable 5,gpt-5.6,GPT 5.6,Opus 5 Review,2.1.255/263/240, old action SHAs) across.github,docs/ci,docs/build, AGENTS.md and the prepare-pr skill. Only the line in Blocking item 1 remains. Old ids elsewhere are product-side (model_registry, test fixtures,model-selection.md), dated RFCs, or "Found in review (GPT 5.6)" attributions. The PR body puts all three out of scope. - Action pin.
git/ref/tags/v1.0.239→ tag object → commit97c53473391bff1901034d4b454b5bac7ab7a029. That commit is "chore: bump Claude Code to 2.1.287".action.ymlinputs and outputs (execution_file,conclusion,session_id,structured_output, …) are the same set at9c5ddab,24dcd50and97c5347. - Adjudicator CLI.
package.jsonand every lockfile entry (main package plus 9 platform binaries) are at2.1.287.ADJ_CLAUDE_CODE_FLOORis2.1.284in bothcodex-review.yml:1049andfork-gpt-review.yml:1060. Thesort -Vcomparison is unchanged. - Readiness triggers. Every
pr-readiness.ymlon.workflow_run.workflowsentry matches a workflowname:at head.CodeQLis the only unmatched entry, and that predates this PR (default setup). Thecheckrun:specs and theopus-pr-/gpt-pr-map atpr-readiness.yml:907-931,1988use the new names. - Fork check-runs. The fork lanes create and finalise check-runs named exactly
Opus 5.5 Review(fork-opus-review.yml:264,1472,1502) andGPT 6.1 Review(fork-gpt-review.yml:253,2182,2217). - Required checks. The PR body says only
PR Readinessis required. Confirmed: branch protectionrequired_status_checks.contextsis empty, and therules/branches/mainruleset requires onlyPR Readiness. Renaming the lanes does not strand the protection. - Secrets and permissions. The diff restricted to non-model lines touches no
secrets.*,permissions:,role-to-assumeoraws-region. Every changed line there is a model id, an action SHA, a display name, or comment text. - No-hardcoded-model rule. The
code-review.yml:186-192check only scanssrc/kiro_crew/**/*.py, and AGENTS.md "Never hardcode a model id" targets product defaults and fallbacks. Every new literal is in.github/workflows/*.yml, docs, a JSON profile or tests, so the rule is respected. Theprepare-pr/profiles/kirocrew.jsonpins mirror CI on purpose and are checked by parity tests. - Tests. I ran the touched contract tests at head (
kiro_crew.__file__resolved into the worktree):test_ai_review_workflows.py,test_review_repair_routing_skill.py,test_prepare_pr_profiles.py,test_prepare_pr_local_review.py,test_pr_readiness_evaluate.py,test_pr_quality_gates.py. Result: 1563 passed. - Mutation 1. Raising
ADJ_CLAUDE_CODE_FLOORto2.1.290incodex-review.ymlturned two tests red:TestAdjudicatorVersionFloor::test_the_committed_pin_satisfies_the_floorand::test_the_lanes_agree_on_the_floor. - Mutation 2. Setting the profile opus model back to
claude-opus-5turnedtest_opus_profile_model_matches_the_ci_workflowandtest_model_pins_agree_with_the_bundled_profilered. - Restored. Both files were restored and
git status --shortwas clean.
Fail-open / fail-closed:
- Floor rise: fail-closed. If the pin lags, the floor step fails, adjudication is skipped, and GPT's blocking verdict stands.
- Unrecognised model id or IAM gap: fail-closed for Opus and GPT (the gates need the
[*-REVIEWED]marker). The same-repo Design, UX and First Principles lanes go red on a model-step error. - Nothing in the diff weakens a gate.
Bot findings
- Design (PASS), note "adjudicator 2.1.287 pin + Sonnet 5.5 fallback have never run": true. On this PR,
review-cliis staged from the BASE commit (2.1.263, which is below the new 2.1.284 floor). Any adjudication on this PR would therefore have failed the floor step closed. GPT found no blocking items, so the stage never ran. This is a post-merge watch item, not a defect. - First Principles (CONCERNS), "repair ladder edit is outside the CI-only Goal": true as a scope observation, not a defect. The edit is to
prepare-pr/SKILL.md:93-94, andSKILL.md:87-89says these preferences "do not change CI models". The description gives a reason for the GPT row but not for removing the Opus/Fable rungs. The change is internally consistent and pinned bytest_family_ladders_preserve_generation_then_capability_fallback. The author should post a disposition: either restore the Fable rungs, or state in the body that local repair routing is deliberately in scope. - Security Scope (CONCERNS):
- "Floor tightening refuses older locks": true and intended. This is the fail-closed path above.
- "Pre-release suffix parsing": not introduced here. The
grep -oE '[0-9]+\.[0-9]+\.[0-9]+'+sort -Vcode (codex-review.yml:1060-1065) is unchanged, and it strips any suffix. - "If branch protection's required-check list is not updated, every PR stops": false. Only
PR Readinessis required (ruleset read above), and readiness resolves same-repo lanes by workflow file.
- Opus 5.5 / GPT 6.1: no findings. I have no disagreement with either.
Follow-ups (non-blocking)
- PR body only (the commit message doesn't carry it, so it won't reach main's squash history): "every lane moves to
claude-code-actionv1.0.239, which bundles 2.1.287 and has no other changes in between" holds for the 14 uses that were on9c5ddab…, but not for the two Security Scope uses (security-scope-review.yml,fork-security-scope-review.yml), which were on24dcd50…(2.1.240). Those also pick up upstream fixes #1702, #1708, #1710, #1713 and #1719. Inputs/outputs are identical across the pins and those lanes runRead,Grep,Globonly, so I don't expect breakage; please correct the sentence. claude-review.yml:52-56andfork-opus-review.yml:24still say branch protection keys on these job names. That predates this PR (onlyPR Readinessis required), but this PR rewrote the adjacent line. A small comment correction would help.gui-user-test.yml:223anddocs/build/gui-user-test.md:93,374still budget at--price-in 15 --price-out 75("Opus list prices"). I could not verify Opus 5.5 list prices. If they are lower, as the body says, the budget gate overstates spend, which is the conservative direction. Update the prices once confirmed.- Watch the first post-merge GPT run that reaches adjudication: it is the first exercise of the 2.1.287 CLI with
us.anthropic.claude-opus-5-5/claude-sonnet-5-5.
buluoray
left a comment
There was a problem hiding this comment.
Approving. Superseding my earlier request-changes: nothing in the logic blocks this PR. The model swap is consistent across every lane, no secret, permission, role or region changed, and the version-floor and profile-pin tests redden when their pins are mutated. Full verification is in my previous review.
Follow-ups (non-blocking):
#16238 renamed the Opus and GPT review lanes. The publish step's fork re-check binds a Stage-2 row by name as well as external_id, so a fork head reviewed before the rename scored green and then re-checked as "Opus 5.5 Review (not started), GPT 6.1 Review (not started)" forever, with only a push able to clear it. Add one LEGACY_LANE_NAMES table (old name -> new name, renamed lanes only). An old-name row answers for its lane only when the head has no bound row under the current name, and it is read exactly like one: red stays red, running holds, and the external_id binding still applies. The hold step and MONITORED_LANES key on run ids and workflow files, so they need no alias. Removal is tracked in #16373.
#16238 renamed the Opus and GPT review lanes. The publish step's fork re-check binds a Stage-2 row by name as well as external_id, so a fork head reviewed before the rename scored green and then re-checked as "Opus 5.5 Review (not started), GPT 6.1 Review (not started)" forever, with only a push able to clear it. Add one LEGACY_LANE_NAMES table (old name -> new name, renamed lanes only). An old-name row answers for its lane only when the head has no bound row under the current name, and it is read exactly like one: red stays red, running holds, and the external_id binding still applies. The hold step and MONITORED_LANES key on run ids and workflow files, so they need no alias. Removal is tracked in #16373.
Rebase onto current main (resolved conflict: keep codeql_pr_runs param added on main while dropping the four legacy-alias params the PR removes). PR #16000 (the only green fork PR with pre-rename rows at review time) closed 2026-10-19. Lowest-numbered open fork PR is now #20034 — all open fork PRs post-date #16238's rename. Removal condition holds.
Rebase onto current main (resolved conflict: keep codeql_pr_runs param added on main while dropping the four legacy-alias params the PR removes). PR #16000 (the only green fork PR with pre-rename rows at review time) closed 2026-10-19. Lowest-numbered open fork PR is now #20034 — all open fork PRs post-date #16238's rename. Removal condition holds.
The publish step's fork re-check carried a LEGACY_LANE_NAMES table so a fork head reviewed before #16238 renamed the Opus and GPT lanes could still score its old-name rows. Every open fork PR that still carries only old-name rows already publishes PR Readiness failure for other reasons, so the alias no longer changes any verdict. Drop the table, the jq fallback that read it, the tests that pinned it and the docs paragraph that explained it.
The publish step's fork re-check carried a LEGACY_LANE_NAMES table so a fork head reviewed before #16238 renamed the Opus and GPT lanes could still score its old-name rows. Every open fork PR that still carries only old-name rows already publishes PR Readiness failure for other reasons, so the alias no longer changes any verdict. Drop the table, the jq fallback that read it, the tests that pinned it and the docs paragraph that explained it.
The publish step's fork re-check carried a LEGACY_LANE_NAMES table so a fork head reviewed before #16238 renamed the Opus and GPT lanes could still score its old-name rows. Every open fork PR that still carries only old-name rows already publishes PR Readiness failure for other reasons, so the alias no longer changes any verdict. Drop the table, the jq fallback that read it, the tests that pinned it and the docs paragraph that explained it.
The publish step's fork re-check carried a LEGACY_LANE_NAMES table so a fork head reviewed before #16238 renamed the Opus and GPT lanes could still score its old-name rows. Every open fork PR that still carries only old-name rows already publishes PR Readiness failure for other reasons, so the alias no longer changes any verdict. Drop the table, the jq fallback that read it, the tests that pinned it and the docs paragraph that explained it.
The publish step's fork re-check carried a LEGACY_LANE_NAMES table so a fork head reviewed before #16238 renamed the Opus and GPT lanes could still score its old-name rows. Every open fork PR that still carries only old-name rows already publishes PR Readiness failure for other reasons, so the alias no longer changes any verdict. Drop the table, the jq fallback that read it, the tests that pinned it and the docs paragraph that explained it.
The publish step's fork re-check carried a LEGACY_LANE_NAMES table so a fork head reviewed before #16238 renamed the Opus and GPT lanes could still score its old-name rows. Every open fork PR that still carries only old-name rows already publishes PR Readiness failure for other reasons, so the alias no longer changes any verdict. Drop the table, the jq fallback that read it, the tests that pinned it and the docs paragraph that explained it.
Problem / Motivation
Goal: Every AI review lane in CI runs on Opus 5.5 or GPT 6.1 Sol, falls back to Sonnet 5.5, and no CI step calls Fable any more.
The review fleet still runs last quarter's models: Opus 5 and GPT 5.6 Sol for the blocking lanes, Fable 5 for Design, UX, First Principles and Security Scope, and Opus 4.8 as every fallback. All of the newer models are authorized in the review account (
us-west-2andus-east-1,us.andglobal.profiles), and the existing IAM grants (anthropic.*,openai.*) already cover them.Why it matters
The new models score higher on the coding benchmarks the vendors publish and cost less per token, most of all for cache reads, which is where an agentic reviewer spends most of its tokens. Fable 5 is the most expensive model in the fleet and Opus 5.5 beats it on most published rows.
Not a goal
src/kiro_crew/model_registry.json, the website registry). Fable stays selectable in the app./ai-review override fable ...command word. It names the Opus lane, not a model, and keeps working unchanged.vars.SHIP_REPORT_BEDROCK_MODEL_IDoverrides ship-report's default and is changed outside this diff.What changed (motivation → approach → change)
claude-opus-5→claude-opus-5-5opus-4-8→sonnet-5-5gpt-5.6-sol→openai.gpt-6.1-solclaude-opus-5→claude-opus-5-5opus-4-8→sonnet-5-5claude-fable-5→claude-opus-5-5opus-4-8→sonnet-5-5opus-5→opus-5-5opus-4-8→sonnet-5-5Opus 5.5 needs Claude Code 2.1.280 and Sonnet 5.5 needs 2.1.284. The pinned
claude-code-actionbundled 2.1.263 (2.1.240 on the Security Scope lanes), so every lane moves toclaude-code-actionv1.0.239, which bundles 2.1.287 and has no other changes in between. The adjudicator installs its CLI from.github/review-cli, so that lockfile pins@anthropic-ai/claude-code2.1.287 andADJ_CLAUDE_CODE_FLOORrises to 2.1.284.@openai/codexstays at 0.150.1: this PR's own GPT lane ranopenai.gpt-6.1-solon Codex v0.150.1 and completed.The two model-named checks are renamed "Opus 5.5 Review" and "GPT 6.1 Review", following the repo rule that a lane carries its model's name. The rename covers the workflows,
pr-readiness.yml, the override rerun table,docs/ci/ci-and-reviews.mdand the tests that pin those names. The lane-named checks (Design, UX, First Principles, Security Scope) keep their names, and only their headings drop "Fable 5".The prepare-pr profile's local reviewers mirror the new CI pins (
gpt-6.1-sol,claude-opus-5.5, tierclaude-sonnet-5.5), and the Review repair routing ladders inSKILL.mddrop Fable. The GPT ladder also moves up a generation: its first two rungs wereGPT 6 -> GPT 5.6 best available variantand are nowGPT 6 Astra -> GPT 6.1 Sol, because the CI lane it repairs now runs GPT 6.1 Sol.docs/build/gui-user-test.mdfollows the workflow's new default model.Backwards compatibility
Compatible: every model is authorized in the review account under the existing IAM grants, and the required check (
PR Readiness) is unchanged.external_id, so the renamed label is display only.fable|gpt|...command words are unchanged (test_ai_review_workflows.py).claude-ai-review/codex-ai-reviewcomment markers, not check names (test_prepare_pr_status.py, test_prepare_pr_supersession.py).Tests
Existing workflow-contract tests are updated to the new model ids, step names and check names.
TestAdjudicatorVersionFloorchecks that the review-cli pin (2.1.287) meets the new floor (2.1.284) and that the lockfile matches.test_opus_profile_model_matches_the_ci_workflowandtest_model_pins_agree_with_the_bundled_profilelock the local reviewer pins to CI.test_family_ladders_preserve_generation_then_capability_fallbackpins the Fable-free repair ladders.Manual verification
Checked with read-only Bedrock calls against the review account:
get-foundation-model-availabilityreportsAUTHORIZEDforclaude-opus-5-5,claude-sonnet-5-5andgpt-6.1-solin both regions, andlist-inference-profilesshows theirus.profiles ACTIVE. The live check is this PR's own lanes, which run the new models. The GPT lane installs.github/review-clifrom the base commit, so on this PR it ran Codex v0.150.1 withopenai.gpt-6.1-soland passed; the adjudicator's new Claude Code pin takes effect once this lands on main.Related Issues
no linked issue: model upgrade requested directly by the maintainer.
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)