Skip to content

ci(review): move every AI review lane to Opus 5.5 and GPT 6.1 Sol - #16238

Merged
bolichen97 merged 1 commit into
mainfrom
ci/review-models-opus55-gpt61
Oct 3, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
ci/review-models-opus55-gpt61

Conversation

@bolichen97

@bolichen97 bolichen97 commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Every AI review lane in CI runs on Opus 5.5 or GPT 6.1 Sol, falls back to Sonnet 5.5, and no CI step calls Fable any more.

The review fleet still runs last quarter's models: Opus 5 and GPT 5.6 Sol for the blocking lanes, Fable 5 for Design, UX, First Principles and Security Scope, and Opus 4.8 as every fallback. All of the newer models are authorized in the review account (us-west-2 and us-east-1, us. and global. profiles), and the existing IAM grants (anthropic.*, openai.*) already cover them.

Why it matters

The new models score higher on the coding benchmarks the vendors publish and cost less per token, most of all for cache reads, which is where an agentic reviewer spends most of its tokens. Fable 5 is the most expensive model in the fleet and Opus 5.5 beats it on most published rows.

Not a goal

  • The product model registry and model picker (src/kiro_crew/model_registry.json, the website registry). Fable stays selectable in the app.
  • The /ai-review override fable ... command word. It names the Opus lane, not a model, and keeps working unchanged.
  • Repo variables. vars.SHIP_REPORT_BEDROCK_MODEL_ID overrides ship-report's default and is changed outside this diff.
  • Historical comments that record which model found a past bug.

What changed (motivation → approach → change)

Lane Primary Fallback
Opus Review + fork twin claude-opus-5 → claude-opus-5-5 opus-4-8 → sonnet-5-5
GPT Review + fork twin gpt-5.6-sol → openai.gpt-6.1-sol (none)
GPT adjudicator ("second verifier") claude-opus-5 → claude-opus-5-5 opus-4-8 → sonnet-5-5
Design, UX, First Principles, Security Scope + fork twins, fix-loop-analysis claude-fable-5 → claude-opus-5-5 opus-4-8 → sonnet-5-5
gui-user-test, issue-summary, issue-triage, ship-report defaults opus-5 → opus-5-5 opus-4-8 → sonnet-5-5

Opus 5.5 needs Claude Code 2.1.280 and Sonnet 5.5 needs 2.1.284. The pinned claude-code-action bundled 2.1.263 (2.1.240 on the Security Scope lanes), so every lane moves to claude-code-action v1.0.239, which bundles 2.1.287 and has no other changes in between. The adjudicator installs its CLI from .github/review-cli, so that lockfile pins @anthropic-ai/claude-code 2.1.287 and ADJ_CLAUDE_CODE_FLOOR rises to 2.1.284. @openai/codex stays at 0.150.1: this PR's own GPT lane ran openai.gpt-6.1-sol on Codex v0.150.1 and completed.

The two model-named checks are renamed "Opus 5.5 Review" and "GPT 6.1 Review", following the repo rule that a lane carries its model's name. The rename covers the workflows, pr-readiness.yml, the override rerun table, docs/ci/ci-and-reviews.md and the tests that pin those names. The lane-named checks (Design, UX, First Principles, Security Scope) keep their names, and only their headings drop "Fable 5".

The prepare-pr profile's local reviewers mirror the new CI pins (gpt-6.1-sol, claude-opus-5.5, tier claude-sonnet-5.5), and the Review repair routing ladders in SKILL.md drop Fable. The GPT ladder also moves up a generation: its first two rungs were GPT 6 -> GPT 5.6 best available variant and are now GPT 6 Astra -> GPT 6.1 Sol, because the CI lane it repairs now runs GPT 6.1 Sol. docs/build/gui-user-test.md follows the workflow's new default model.

Backwards compatibility

Compatible: every model is authorized in the review account under the existing IAM grants, and the required check (PR Readiness) is unchanged.

  • Reader: .github/workflows/pr-readiness.yml:on.workflow_run.workflows -- CI -- renamed here. An open PR whose head predates this still runs lanes named "Opus 5 Review" / "GPT 5.6 Review"; their completion no longer triggers readiness, so that PR re-evaluates on its next CI / Fast Gate event or the readiness sweep, and fully after a rebase.
  • Reader: .github/workflows/pr-readiness.yml:workflow_specs -- CI -- same-repo lanes are resolved by workflow file and fork lanes by external_id, so the renamed label is display only.
  • Reader: .github/workflows/ai-review-human-override.yml:rerun_reviewer -- CI -- table renamed here; the fable|gpt|... command words are unchanged (test_ai_review_workflows.py).
  • Reader: src/kiro_crew/builtin_skills/kirocrew-dev/prepare-pr/scripts/_review_contract.py:marker bindings -- chat|subagent -- keyed on the claude-ai-review / codex-ai-review comment markers, not check names (test_prepare_pr_status.py, test_prepare_pr_supersession.py).
  • Reader: src/kiro_crew/monitoring/pull_request.py:red-check identity -- chat|cron -- keys on whatever name the check carries, no hard-coded lane names (test_irq.py).

Tests

Existing workflow-contract tests are updated to the new model ids, step names and check names. TestAdjudicatorVersionFloor checks that the review-cli pin (2.1.287) meets the new floor (2.1.284) and that the lockfile matches. test_opus_profile_model_matches_the_ci_workflow and test_model_pins_agree_with_the_bundled_profile lock the local reviewer pins to CI. test_family_ladders_preserve_generation_then_capability_fallback pins the Fable-free repair ladders.

Manual verification

Checked with read-only Bedrock calls against the review account: get-foundation-model-availability reports AUTHORIZED for claude-opus-5-5, claude-sonnet-5-5 and gpt-6.1-sol in both regions, and list-inference-profiles shows their us. profiles ACTIVE. The live check is this PR's own lanes, which run the new models. The GPT lane installs .github/review-cli from the base commit, so on this PR it ran Codex v0.150.1 with openai.gpt-6.1-sol and passed; the adjudicator's new Claude Code pin takes effect once this lands on main.

Related Issues

no linked issue: model upgrade requested directly by the maintainer.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@bolichen97
bolichen97 requested a review from a team as a code owner October 2, 2026 19:30
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 2, 2026
@github-actions

github-actions Bot commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of a20d9e1e068fa88f76fb8cbd524e210086c1bc44 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

The GPT adjudicator's Claude Code 2.1.287 pin and its Sonnet 5.5 fallback have never run. Check the first adjudicator run on main after merge.

[DESIGN-REVIEWED] a20d9e1

Description read

[DESCRIPTION-READ] 81eb15a432b5b8de46cbabc822f507189fd6ae20b59565495388b7fba55d7689

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — 🟡 CONCERNS

Premise-level review of a20d9e1e068fa88f76fb8cbd524e210086c1bc44 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: CONCERNS

The prepare-pr repair ladder drops Fable 5.1 and Fable 5, but the Goal only covers CI. This ladder is a local skill preference, and the skill says CI models do not set it.

Not justified as shipped

  • Item 10 — oversized: The Goal is "no CI step calls Fable any more", and Not a goal says "Fable stays selectable in the app". The ladder this hunk edits sits under the line "These prose family preferences do not change CI models" (SKILL.md, added in refactor: slim built-in prompts and route review repairs #10293). The diff deletes "Fable 5.1", "Fable 5" from test_family_ladders_preserve_generation_then_capability_fallback. For the GPT ladder the description's reason is "the CI lane it repairs now runs GPT 6.1 Sol", but the skill says that does not decide the ladder. For the Opus ladder it gives no reason. Every CI item reaches the Goal without this one.
    Subtraction: in prepare-pr/SKILL.md and test_review_repair_routing_skill.py, put back the Fable 5.1 -> Fable 5 -> rungs and GPT 6 -> GPT 5.6 best available variant. Keep only the lane-label rename (GPT 6.1 review lane).
    Clears when: the ladder rungs are restored, or a human amends the Goal to cover local repair routing.

What this change ships

Inventory (10 items) — 9 justified

Intent: the maintainer wants every CI review lane on newer, cheaper models with Fable gone from CI. This is an ADDITION (a model upgrade).

  1. Opus Review lane, its fork twin and the GPT adjudicator run Opus 5.5 — justified
  2. GPT Review lane and its fork twin run GPT 6.1 Sol — justified
  3. Design, UX, First Principles, Security Scope and fix-loop-analysis move from Fable 5 to Opus 5.5 — justified
  4. Every overload fallback moves from Opus 4.8 to Sonnet 5.5 — justified
  5. The two model-named checks are renamed, and the readiness triggers and override table follow — justified
  6. Lane comment headings and log text show "Opus 5.5" instead of "Fable 5" — justified
  7. claude-code-action is pinned to v1.0.239, review-cli to Claude Code 2.1.287, and the adjudicator floor rises to 2.1.284 — justified
  8. Defaults for gui-user-test, issue-summary, issue-triage and ship-report move to Opus 5.5 → Sonnet 5.5 — justified
  9. prepare-pr local reviewer pins follow the new CI models — justified
  10. Local repair routing stops preferring Fable for Opus-lane fixes and re-ranks the GPT rungs — oversized, outside the CI-only Goal

[FIRST-PRINCIPLES-REVIEWED] a20d9e1

Description read

[DESCRIPTION-READ] a0230c642b670db10c8325d4d59f3b885a4934fb9c843aa7aa784b9a1fc06917

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed a20d9e1e068fa88f76fb8cbd524e210086c1bc44 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] a20d9e1

Verdict parsed from the review's SHA-scoped output markers for commit a20d9e1e068fa88f76fb8cbd524e210086c1bc44.

False positive or not applicable? A repository writer can comment:
/ai-review override fable a20d9e1e068fa88f76fb8cbd524e210086c1bc44: <one-sentence reason>

@github-actions

github-actions Bot commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of a20d9e1e068fa88f76fb8cbd524e210086c1bc44 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] a20d9e1

False positive or not applicable? A repository writer can comment:
/ai-review override gpt a20d9e1e068fa88f76fb8cbd524e210086c1bc44: <one-sentence reason>

@github-actions

github-actions Bot commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

Security Scope Review -- 🟡 CONCERNS

Which legitimate operations does a20d9e1e068fa88f76fb8cbd524e210086c1bc44 newly refuse? Updated in place on each push. A script-confirmed regression blocks; the model proposes candidates and never decides a refusal.

Verdict basis: the reviewer's own judgement; no confirmed regression.

Security scope review -- adjudicated candidates

  • macos-latest (platform posix): 9 classified, 0 newly refused, 9 still allowed.
  • ubuntu-latest (platform posix): 9 classified, 0 newly refused, 9 still allowed.
  • windows-latest (platform windows): 8 classified, 0 newly refused, 8 still allowed.

No candidate flipped from allowed to refused.

Scope-Verdict: CONCERNS

TIGHTENING: The only control this diff makes stricter is the CI version floor on the adjudicator's Claude Code CLI. It refuses to run the Opus 5.5 adjudication stage on a Claude Code CLI too old for the new primary and fallback models. The hunk in codex-review.yml and fork-gpt-review.yml: - ADJ_CLAUDE_CODE_FLOOR: "2.1.255" / + ADJ_CLAUDE_CODE_FLOOR: "2.1.284" ("Opus 5.5 needs 2.1.280 and the Sonnet 5.5 fallback 2.1.284"). Nothing under the tool-call deny composite (path fence, sensitive-command tier, exfiltration shapes, rule catalog) changes. The rest of the diff renames check-runs from "Opus 5"/"GPT 5.6" to "Opus 5.5"/"GPT 6.1", swaps model ids, moves the claude-code-action pin to 97c53473… (v1.0.239) and bumps the lockfile pin to 2.1.287.

ADJUDICATED: 12 candidates, 0 confirmed newly-refused (no verdicts had come back when this review was written; they are classifier rows, not claimed results)

PLATFORM COVERAGE: The diff adds no path, separator, interpreter or env-var assumption to the deny composite. The candidates still cover the operations around this change in every spelling:

  • any: the gh run rerun --failed and --job forms that the edited prepare-pr SKILL.md prescribes, the gh api …/check-runs?check_name=Opus%205.5%20Review query against the renamed check, gh pr checks --required, and the gh workflow run gui-user-test.yml dispatch with the new default model.
  • posix: the lockfile regeneration that package.json prescribes, npm ci --prefix, the node_modules/.bin/claude --version floor check, and the workflow contract tests.
  • windows: the same lockfile regeneration with \ separators, the claude.cmd shim version check, and py -m pytest with backslash paths.

The CI-side floor runs only on ubuntu-latest. It assumes a posix $RUNNER_TEMP/review-cli/node_modules/.bin/claude path, and no Windows runner exists for it to cover.

UNADJUDICATED: The floor check is a CI version gate in the "adjudicator floor" step of codex-review.yml and fork-gpt-review.yml. Its call path: npm ci from .github/review-cli installs the locked @anthropic-ai/claude-code → the step runs "$bin" --version on $RUNNER_TEMP/review-cli/node_modules/.bin/claude → it compares the result with ADJ_CLAUDE_CODE_FLOOR → if older, it fails at configuration time, before the "Opus 5.5 adjudication" step can run. The classifier cannot judge this gate. These are the inputs I would have submitted:

  • The lock at this head: pin 2.1.287 against floor 2.1.284, which should pass.
  • A downgraded lock: any 2.1.255–2.1.283 lock that satisfied the old floor and is now refused. This is intended, but it means Opus 4.8 can no longer serve as an emergency fallback without a matching floor edit.
  • A pre-release suffix: a version such as 2.1.284-beta.1. Whether the comparison treats it as at or below the floor depends on version-parsing code that this diff does not show.

Model ids and check names also moved (us.anthropic.claude-sonnet-5-5 as fallback, required checks "Opus 5.5 Review"/"GPT 6.1 Review"). If branch protection's required-check list is not updated alongside this change, every PR stops at readiness. The classifier cannot reach that admission surface either.

[SCOPE-REVIEWED] a20d9e1

@bolichen97
bolichen97 force-pushed the ci/review-models-opus55-gpt61 branch from dbe8181 to 839de29 Compare October 2, 2026 19:46
@bolichen97

Copy link
Copy Markdown
Contributor Author
  • fixed: @openai/codex reverted to 0.150.1 span=49ab56da87b9

No Codex release is needed for openai.gpt-6.1-sol: this PR's own GPT 6.1 Review run logged OpenAI Codex v0.150.1 with model: openai.gpt-6.1-sol and completed.
839de29 reverts the pin to 0.150.1 and regenerates the lockfile; only the @anthropic-ai/claude-code 2.1.263 -> 2.1.287 bump remains, which the 2.1.284 adjudicator floor requires.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • rebutted: the repair ladder change is part of retiring Fable from the review loop span=8eb67f720054

The maintainer's request was to retire Fable from the review fleet entirely, not only from CI job steps; the Goal sentence names the CI side of that same request.
The Review repair routing table exists only to repair findings from these CI review lanes, and its rows are keyed by lane name: the GPT row has to change here regardless, because the lane is renamed "GPT 6.1 Review".
Keeping "Fable 5.1 -> Fable 5" as the first repair rung would route every Opus-lane repair to the model this PR removes from the fleet, which contradicts the request it implements.

@bolichen97

Copy link
Copy Markdown
Contributor Author

Frontend Tests (1)/(2) are red on this head for reasons this PR does not touch (it changes no website/ file):

I'll rebase onto main once one of those lands and let CI re-run.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • rebutted: dropping Fable from the repair ladder is the maintainer's explicit scope span=db9ad6192044

The maintainer asked for Fable to be retired from the review fleet entirely, and asked for this in the same request as the model swap; the repair ladder is the one remaining place the review loop selects Fable.
The ladder routes repairs for exactly these CI lanes and is keyed by their names, so it is part of the review fleet's configuration, not the app's model picker that Not a goal keeps.
The pinned order in the test is the order this PR changes on purpose; the test is updated to the new order in the same diff, so nothing silently reverses it.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • rebutted: the four digest/test jobs follow the maintainer's "every fallback is Sonnet 5.5" instruction span=561a05b4e196

The maintainer asked that every CI fallback move to Sonnet 5.5; gui-user-test, issue-summary, issue-triage and ship-report are the remaining CI jobs whose default chain ended on Opus 4.8.
Each change is a default-string edit with no new mechanism, and all four run under roles whose existing anthropic.* grant already covers the new ids.

@bolichen97

Copy link
Copy Markdown
Contributor Author

Rebased onto main to pick up the frontend test fix (#16227); CI re-running.

@bolichen97
bolichen97 force-pushed the ci/review-models-opus55-gpt61 branch from 839de29 to 6ab3444 Compare October 2, 2026 23:44
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 2, 2026
@bolichen97
bolichen97 force-pushed the ci/review-models-opus55-gpt61 branch from 6ab3444 to 7141b0c Compare October 3, 2026 00:23
@bolichen97

Copy link
Copy Markdown
Contributor Author
  • rebutted span=b0c3a85a5ab8

Same item as span db9ad6192044 on the previous head, answered there. The owner asked to retire Fable from the review loop entirely, which covers the local repair ladder that mirrors the CI lanes, not only the CI steps. Keeping Fable at the head of the Opus repair ladder would route review repairs to a model the CI lanes no longer use. The Not-a-goal line is about the product model registry and picker, which this PR does not touch; Fable stays selectable in the app.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • fixed: declared in the description span=987aeb9993e2

The PR description now states the GPT ladder change explicitly: its first two rungs move from GPT 6 -> GPT 5.6 best available variant to GPT 6 Astra -> GPT 6.1 Sol, because the CI lane it repairs now runs GPT 6.1 Sol.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • fixed in 7141b0c span=e776885bcab4

docs/build/gui-user-test.md now names us.anthropic.claude-opus-5-5 as the workflow default (line 92) and in the reproduction command (line 374), matching gui-user-test.yml.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: passed Eligible automated validation passed for the current revision labels Oct 3, 2026
@bolichen97
bolichen97 force-pushed the ci/review-models-opus55-gpt61 branch from 7141b0c to a20d9e1 Compare October 3, 2026 02:11
@bolichen97

Copy link
Copy Markdown
Contributor Author

Rebased onto main (9a59174). Head 7141b0c lost its Security Scope check to the per-head floor: two earlier runs failed on the shared GitHub App rate limit, and the third run (CONCERNS, 0 newly refused) could not clear that. A new head re-measures. First Principles items on 7141b0c were answered in 5964438763 and 5964438901.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 3, 2026
@bolichen97

Copy link
Copy Markdown
Contributor Author
  • rebut span=bac1a4b2a45a

Same item as spans b0c3a85a5ab8 and ead51e86a02b on earlier heads (comments 5963580822, 5964438763). The owner asked to retire Fable from the whole review loop ("把 fable 全面下架"), not only from CI steps. The repair ladder picks the model that repairs review findings, so Fable rungs there keep Fable in that loop. The GPT rungs follow the same loop: they now name the models the review fleet runs. ## Not a goal keeps Fable in the product model registry, which this PR does not touch. Widening the Goal text is the owner's call; until then this stands rebutted.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 3, 2026
@bolichen97
bolichen97 enabled auto-merge (squash) October 3, 2026 03:31

@buluoray buluoray left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary

The model swap itself is correct and consistent. Every --model/--fallback-model/model =/MODEL_IDS default under .github/ now names claude-opus-5-5, claude-sonnet-5-5 or openai.gpt-6.1-sol. All 16 claude-code-action uses are pinned to 97c53473…, which is the v1.0.239 tag. The readiness triggers, the fork check-run names, the override table and the docs all use the new lane names. No secret, permission, role or region changed. Auto-merge is on, so I am applying the stricter bar: one stale workflow header still names the old model, and that is a one-line fix. One inaccurate PR-body sentence is listed as a follow-up.

Blocking

  1. .github/workflows/codex-review.yml:3 still says # Second AI reviewer using OpenAI GPT-5.6 Sol via Amazon Bedrock (Mantle). The workflow is now name: GPT 6.1 Review (line 1) and runs model = "openai.gpt-6.1-sol" (line 235). This PR edited lines 1 and 9 of the same header block but missed this one. It is the only old-model description left under .github/. Line 1023 (#9216 … Opus 4.8's 2.1.255 floor) and release.yml:304 are historical incident notes, so they are fine.

This is the only blocker, and with auto-merge on it would otherwise ship as-is.

What I verified

  • Old ids. I grepped the head tree for old ids (claude-opus-5 not followed by -5, opus-4-8, Opus 4.8, fable-5, Fable 5, gpt-5.6, GPT 5.6, Opus 5 Review, 2.1.255/263/240, old action SHAs) across .github, docs/ci, docs/build, AGENTS.md and the prepare-pr skill. Only the line in Blocking item 1 remains. Old ids elsewhere are product-side (model_registry, test fixtures, model-selection.md), dated RFCs, or "Found in review (GPT 5.6)" attributions. The PR body puts all three out of scope.
  • Action pin. git/ref/tags/v1.0.239 → tag object → commit 97c53473391bff1901034d4b454b5bac7ab7a029. That commit is "chore: bump Claude Code to 2.1.287". action.yml inputs and outputs (execution_file, conclusion, session_id, structured_output, …) are the same set at 9c5ddab, 24dcd50 and 97c5347.
  • Adjudicator CLI. package.json and every lockfile entry (main package plus 9 platform binaries) are at 2.1.287. ADJ_CLAUDE_CODE_FLOOR is 2.1.284 in both codex-review.yml:1049 and fork-gpt-review.yml:1060. The sort -V comparison is unchanged.
  • Readiness triggers. Every pr-readiness.yml on.workflow_run.workflows entry matches a workflow name: at head. CodeQL is the only unmatched entry, and that predates this PR (default setup). The checkrun: specs and the opus-pr-/gpt-pr- map at pr-readiness.yml:907-931,1988 use the new names.
  • Fork check-runs. The fork lanes create and finalise check-runs named exactly Opus 5.5 Review (fork-opus-review.yml:264,1472,1502) and GPT 6.1 Review (fork-gpt-review.yml:253,2182,2217).
  • Required checks. The PR body says only PR Readiness is required. Confirmed: branch protection required_status_checks.contexts is empty, and the rules/branches/main ruleset requires only PR Readiness. Renaming the lanes does not strand the protection.
  • Secrets and permissions. The diff restricted to non-model lines touches no secrets.*, permissions:, role-to-assume or aws-region. Every changed line there is a model id, an action SHA, a display name, or comment text.
  • No-hardcoded-model rule. The code-review.yml:186-192 check only scans src/kiro_crew/**/*.py, and AGENTS.md "Never hardcode a model id" targets product defaults and fallbacks. Every new literal is in .github/workflows/*.yml, docs, a JSON profile or tests, so the rule is respected. The prepare-pr/profiles/kirocrew.json pins mirror CI on purpose and are checked by parity tests.
  • Tests. I ran the touched contract tests at head (kiro_crew.__file__ resolved into the worktree): test_ai_review_workflows.py, test_review_repair_routing_skill.py, test_prepare_pr_profiles.py, test_prepare_pr_local_review.py, test_pr_readiness_evaluate.py, test_pr_quality_gates.py. Result: 1563 passed.
  • Mutation 1. Raising ADJ_CLAUDE_CODE_FLOOR to 2.1.290 in codex-review.yml turned two tests red: TestAdjudicatorVersionFloor::test_the_committed_pin_satisfies_the_floor and ::test_the_lanes_agree_on_the_floor.
  • Mutation 2. Setting the profile opus model back to claude-opus-5 turned test_opus_profile_model_matches_the_ci_workflow and test_model_pins_agree_with_the_bundled_profile red.
  • Restored. Both files were restored and git status --short was clean.

Fail-open / fail-closed:

  • Floor rise: fail-closed. If the pin lags, the floor step fails, adjudication is skipped, and GPT's blocking verdict stands.
  • Unrecognised model id or IAM gap: fail-closed for Opus and GPT (the gates need the [*-REVIEWED] marker). The same-repo Design, UX and First Principles lanes go red on a model-step error.
  • Nothing in the diff weakens a gate.

Bot findings

  • Design (PASS), note "adjudicator 2.1.287 pin + Sonnet 5.5 fallback have never run": true. On this PR, review-cli is staged from the BASE commit (2.1.263, which is below the new 2.1.284 floor). Any adjudication on this PR would therefore have failed the floor step closed. GPT found no blocking items, so the stage never ran. This is a post-merge watch item, not a defect.
  • First Principles (CONCERNS), "repair ladder edit is outside the CI-only Goal": true as a scope observation, not a defect. The edit is to prepare-pr/SKILL.md:93-94, and SKILL.md:87-89 says these preferences "do not change CI models". The description gives a reason for the GPT row but not for removing the Opus/Fable rungs. The change is internally consistent and pinned by test_family_ladders_preserve_generation_then_capability_fallback. The author should post a disposition: either restore the Fable rungs, or state in the body that local repair routing is deliberately in scope.
  • Security Scope (CONCERNS):
    • "Floor tightening refuses older locks": true and intended. This is the fail-closed path above.
    • "Pre-release suffix parsing": not introduced here. The grep -oE '[0-9]+\.[0-9]+\.[0-9]+' + sort -V code (codex-review.yml:1060-1065) is unchanged, and it strips any suffix.
    • "If branch protection's required-check list is not updated, every PR stops": false. Only PR Readiness is required (ruleset read above), and readiness resolves same-repo lanes by workflow file.
  • Opus 5.5 / GPT 6.1: no findings. I have no disagreement with either.

Follow-ups (non-blocking)

  • PR body only (the commit message doesn't carry it, so it won't reach main's squash history): "every lane moves to claude-code-action v1.0.239, which bundles 2.1.287 and has no other changes in between" holds for the 14 uses that were on 9c5ddab…, but not for the two Security Scope uses (security-scope-review.yml, fork-security-scope-review.yml), which were on 24dcd50… (2.1.240). Those also pick up upstream fixes #1702, #1708, #1710, #1713 and #1719. Inputs/outputs are identical across the pins and those lanes run Read,Grep,Glob only, so I don't expect breakage; please correct the sentence.
  • claude-review.yml:52-56 and fork-opus-review.yml:24 still say branch protection keys on these job names. That predates this PR (only PR Readiness is required), but this PR rewrote the adjacent line. A small comment correction would help.
  • gui-user-test.yml:223 and docs/build/gui-user-test.md:93,374 still budget at --price-in 15 --price-out 75 ("Opus list prices"). I could not verify Opus 5.5 list prices. If they are lower, as the body says, the budget gate overstates spend, which is the conservative direction. Update the prices once confirmed.
  • Watch the first post-merge GPT run that reaches adjudication: it is the first exercise of the 2.1.287 CLI with us.anthropic.claude-opus-5-5 / claude-sonnet-5-5.

@buluoray buluoray left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving. Superseding my earlier request-changes: nothing in the logic blocks this PR. The model swap is consistent across every lane, no secret, permission, role or region changed, and the version-floor and profile-pin tests redden when their pins are mutated. Full verification is in my previous review.

Follow-ups (non-blocking):

  • .github/workflows/codex-review.yml:3 still says GPT-5.6 Sol in the header comment.
  • PR body: "no other changes in between" doesn't hold for the two Security Scope lanes, which were on 24dcd50… and also pick up upstream fixes #1702, #1708, #1710, #1713 and #1719.

@bolichen97
bolichen97 merged commit d10becf into main Oct 3, 2026
99 checks passed
@bolichen97
bolichen97 deleted the ci/review-models-opus55-gpt61 branch October 3, 2026 03:49
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 3, 2026
bolichen97 added a commit that referenced this pull request Oct 4, 2026
#16238 renamed the Opus and GPT review lanes. The publish step's fork
re-check binds a Stage-2 row by name as well as external_id, so a fork
head reviewed before the rename scored green and then re-checked as
"Opus 5.5 Review (not started), GPT 6.1 Review (not started)" forever,
with only a push able to clear it.

Add one LEGACY_LANE_NAMES table (old name -> new name, renamed lanes
only). An old-name row answers for its lane only when the head has no
bound row under the current name, and it is read exactly like one: red
stays red, running holds, and the external_id binding still applies.
The hold step and MONITORED_LANES key on run ids and workflow files, so
they need no alias. Removal is tracked in #16373.
bolichen97 added a commit that referenced this pull request Oct 4, 2026
#16238 renamed the Opus and GPT review lanes. The publish step's fork
re-check binds a Stage-2 row by name as well as external_id, so a fork
head reviewed before the rename scored green and then re-checked as
"Opus 5.5 Review (not started), GPT 6.1 Review (not started)" forever,
with only a push able to clear it.

Add one LEGACY_LANE_NAMES table (old name -> new name, renamed lanes
only). An old-name row answers for its lane only when the head has no
bound row under the current name, and it is read exactly like one: red
stays red, running holds, and the external_id binding still applies.
The hold step and MONITORED_LANES key on run ids and workflow files, so
they need no alias. Removal is tracked in #16373.
CrysisDeu pushed a commit that referenced this pull request Oct 25, 2026
Rebase onto current main (resolved conflict: keep codeql_pr_runs param
added on main while dropping the four legacy-alias params the PR removes).

PR #16000 (the only green fork PR with pre-rename rows at review time)
closed 2026-10-19. Lowest-numbered open fork PR is now #20034 — all open
fork PRs post-date #16238's rename. Removal condition holds.
CrysisDeu pushed a commit that referenced this pull request Oct 25, 2026
Rebase onto current main (resolved conflict: keep codeql_pr_runs param
added on main while dropping the four legacy-alias params the PR removes).

PR #16000 (the only green fork PR with pre-rename rows at review time)
closed 2026-10-19. Lowest-numbered open fork PR is now #20034 — all open
fork PRs post-date #16238's rename. Removal condition holds.
bolichen97 added a commit that referenced this pull request Nov 7, 2026
The publish step's fork re-check carried a LEGACY_LANE_NAMES table so a
fork head reviewed before #16238 renamed the Opus and GPT lanes could
still score its old-name rows. Every open fork PR that still carries
only old-name rows already publishes PR Readiness failure for other
reasons, so the alias no longer changes any verdict. Drop the table,
the jq fallback that read it, the tests that pinned it and the docs
paragraph that explained it.
bolichen97 added a commit that referenced this pull request Nov 7, 2026
The publish step's fork re-check carried a LEGACY_LANE_NAMES table so a
fork head reviewed before #16238 renamed the Opus and GPT lanes could
still score its old-name rows. Every open fork PR that still carries
only old-name rows already publishes PR Readiness failure for other
reasons, so the alias no longer changes any verdict. Drop the table,
the jq fallback that read it, the tests that pinned it and the docs
paragraph that explained it.
bolichen97 added a commit that referenced this pull request Nov 7, 2026
The publish step's fork re-check carried a LEGACY_LANE_NAMES table so a
fork head reviewed before #16238 renamed the Opus and GPT lanes could
still score its old-name rows. Every open fork PR that still carries
only old-name rows already publishes PR Readiness failure for other
reasons, so the alias no longer changes any verdict. Drop the table,
the jq fallback that read it, the tests that pinned it and the docs
paragraph that explained it.
bolichen97 added a commit that referenced this pull request Nov 7, 2026
The publish step's fork re-check carried a LEGACY_LANE_NAMES table so a
fork head reviewed before #16238 renamed the Opus and GPT lanes could
still score its old-name rows. Every open fork PR that still carries
only old-name rows already publishes PR Readiness failure for other
reasons, so the alias no longer changes any verdict. Drop the table,
the jq fallback that read it, the tests that pinned it and the docs
paragraph that explained it.
bolichen97 added a commit that referenced this pull request Nov 8, 2026
The publish step's fork re-check carried a LEGACY_LANE_NAMES table so a
fork head reviewed before #16238 renamed the Opus and GPT lanes could
still score its old-name rows. Every open fork PR that still carries
only old-name rows already publishes PR Readiness failure for other
reasons, so the alias no longer changes any verdict. Drop the table,
the jq fallback that read it, the tests that pinned it and the docs
paragraph that explained it.
bolichen97 added a commit that referenced this pull request Nov 8, 2026
The publish step's fork re-check carried a LEGACY_LANE_NAMES table so a
fork head reviewed before #16238 renamed the Opus and GPT lanes could
still score its old-name rows. Every open fork PR that still carries
only old-name rows already publishes PR Readiness failure for other
reasons, so the alias no longer changes any verdict. Drop the table,
the jq fallback that read it, the tests that pinned it and the docs
paragraph that explained it.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants