Skip to content

fix(ci): turn off web search in the GPT review lanes - #16161

Merged
bolichen97 merged 1 commit into
kirodotdev:mainfrom
amadsalmon:fix/gpt-review-disable-web-search
Oct 18, 2026
Merged

bolichen97 merged 1 commit into
kirodotdev:mainfrom
amadsalmon:fix/gpt-review-disable-web-search

Conversation

@amadsalmon

@amadsalmon amadsalmon commented Oct 2, 2026 •

Copy link
Copy Markdown
Member

Problem / Motivation

Goal: The GPT review check (GPT 6.1 Review, named GPT 5.6 Review when this was filed) reaches a verdict even when its reviewer wants to search the web, instead of failing with "review incomplete".

Both GPT review lanes write codex's config.toml with no web_search key, and codex 0.150.1 reads an absent key as cached. So the reviewer is offered a web-search tool that the lanes' Bedrock identity is not allowed to use. When the reviewer searches, the call is refused (Access denied: web search is not authorized for this identity), the stream ends, and the pass fails closed with no verdict. #15968 hit this on three heads in a row, and #15200 and #15967 hit it too.

Why it matters

GPT 6.1 Review is a required check, so a PR whose reviewer decides to search cannot merge until a maintainer overrides the lane. A plain rerun fails the same way. It hits same-repo and fork PRs alike.

Not a goal

  • Giving the review lanes a web search they are allowed to use.
  • Changing anything else either lane passes to codex: model, reasoning effort, sandbox mode, shell environment policy.
  • The Opus adjudication step, which runs Claude Code with its own tool list.

What changed (motivation → approach → change)

GPT 6.1 Review no longer fails when its reviewer tries to search the web. Both lanes now write web_search = "disabled" into codex's config, so codex never offers the reviewer a search tool, and the lanes' Bedrock identity has no search call to be refused.

disabled is the only mode for which codex builds no web-search tool. The key goes at the top level of the TOML in codex-review.yml and fork-gpt-review.yml. In the fork lane it sits above the first table header, [model_providers.bedrock-signer], so TOML does not read it as part of a table.

What codex does with the key, read from the source at the pinned rust-v0.150.1 tag:

Behavior Where
web_search is a top-level config key: "disabled, cached, indexed, or live" codex-rs/config/src/config_toml.rs
An absent key becomes cached: resolve_web_search_mode(&cfg, &features).unwrap_or(WebSearchMode::Cached) codex-rs/core/src/config/mod.rs
codex allows cached for every provider, so it offers the tool under Bedrock too, including the fork lane's bedrock-signer proxy provider resolve_web_search_mode_for_turn, same file
disabled builds no tool: Some(WebSearchMode::Disabled) | None => return None codex-rs/core/src/tools/hosted_spec.rs

Backwards compatibility

Compatible: the lanes lose only a tool their identity could never use; each review still runs the same model, prompts and sandbox.

Tests

test/test_ai_review_workflows_credential_isolation.py: TestGptLanesOfferTheModelNoWebSearch runs each lane's "Configure the review CLI for Amazon Bedrock" step and parses the config.toml it writes with tomllib. It fails when the key is absent, holds another mode, or lands inside a table. The helper that runs that step takes the workflow name, so the existing fork-lane config tests share it.

The test catches the bug: with the workflow change reverted, both cases fail; with the key moved under [shell_environment_policy], the fork case fails.

Manual verification

N/A for a live run on this PR. The fork lane runs its workflow from main, so this PR's own GPT 6.1 Review still uses the old config and can fail the same way. The same-repo lane runs the workflow from the PR's branch, so same-repo PRs get the fix once their branch includes it. The codex behavior above is read from the source at the version .github/review-cli/package.json pins.

Related Issues

Fixes #16142

Pattern harvest

Rule candidate: lint
Pattern: a review lane's staged CLI config leaves a capability setting (sandbox mode, web search) to the CLI's default, so the default decides what the reviewer can do.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@amadsalmon
amadsalmon requested a review from a team as a code owner October 2, 2026 14:17
@amadsalmon
amadsalmon requested a review from bolichen97 October 2, 2026 14:17
@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: action required A blocking check or review needs attention labels Oct 2, 2026
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 7, 2026
@github-actions

github-actions Bot commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5, fork) — ✅ PASS

Design-level review of 1253f52d5a726980a8c807622d0ca2823fd3860a via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Check that a later run of the same-repo GPT 6.1 Review on this branch reaches a verdict with web_search = "disabled" set; the fork lane only picks it up after merge.

[DESIGN-REVIEWED] 1253f52

Description read

[DESCRIPTION-READ] c2c546388155f720acaf287b06427c5bbc8d25992b2df2c4d9186f4a6984dca2

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5, fork) — ✅ PASS

Premise-level review of 1253f52d5a726980a8c807622d0ca2823fd3860a via the fork AI-review pipeline — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

Confirm that web_search = "disabled" is honoured by codex 0.150.1. The description cites the source for this but did not run it.

What this change ships

Inventory (3 items) — 3 justified

Intent: let the required GPT review check reach a verdict instead of failing with "review incomplete" when the reviewer tries to search the web. This is a FIX (#16142).

  1. The same-repo GPT review lane no longer offers the reviewer a web-search tool. — justified
  2. The fork GPT review lane no longer offers the reviewer a web-search tool. The key sits at the top level, above the first table. — justified
  3. The test helper that runs the config step now takes the workflow name. Its two callers pass different values, and a new test checks the key for both lanes. — justified

[FIRST-PRINCIPLES-REVIEWED] 1253f52

Description read

[DESCRIPTION-READ] 64f29f053f152074e40aa68bbbb672ef37762b2ff9874e7b15ced6d44f7496bd

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review (fork) — ✅ no blocking findings

Reviewed 1253f52d5a726980a8c807622d0ca2823fd3860a via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 1253f52

@github-actions

github-actions Bot commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review (fork) — ✅ no blocking findings

Reviewed 1253f52d5a726980a8c807622d0ca2823fd3860a via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 1253f52

@amadsalmon

amadsalmon commented Oct 7, 2026 •

Copy link
Copy Markdown
Member Author

Status for maintainers (updated in place). Current head: 1253f52d5a, Amad's one commit rebased onto main 1016b01054.

The two "Update branch" merges took the PR to 3 commits, over PR Hygiene's limit of 2, so the commit was rebased onto main. The diff is unchanged: the same 6 + 6 workflow lines and test change. Locally, the 1,509 tests in test/test_ai_review_workflows*.py and every Backend Lint step pass.

Design and First Principles both ask to confirm codex 0.150.1 accepts web_search = "disabled". A same-repo GPT run can only happen after merge, so I checked it locally with the pinned CLI (npm install @openai/codex@0.150.1, codex --version prints codex-cli 0.150.1). I extracted the exact config.toml each workflow writes and loaded it with CODEX_HOME=<dir> codex mcp list:

Config Result
codex-review.yml's heredoc (4 lines) exit 0
fork-gpt-review.yml's heredoc (53 lines, key above [model_providers.bedrock-signer]) exit 0
Control: web_search = "bogus" exit 1: unknown variant `bogus`, expected one of `disabled`, `cached`, `indexed`, `live`

So disabled is a valid value, and an invalid one fails loudly at startup rather than discarding the file.

CI on 1253f52d5a is green: 87 checks pass, including PR Hygiene, and every AI lane passed with no blocking findings, GPT 6.1 included, so no override is needed on this head. PR Readiness is still computing.

Left for a maintainer: an approving review and the merge. "Update with rebase" keeps the branch at one commit if it goes stale.

Sent by an AI agent on behalf of Amad

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 7, 2026
bolichen97
bolichen97 previously approved these changes Oct 7, 2026

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving: the diff is two web_search = "disabled" lines plus a TOML-parsing test. It only removes a tool the lanes' Bedrock identity could never call. Reran the two timed-out checks.

@github-actions github-actions Bot added the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Oct 9, 2026
@amadsalmon
amadsalmon force-pushed the fix/gpt-review-disable-web-search branch from 450480f to 430b03e Compare October 9, 2026 18:19
@github-actions github-actions Bot removed allow-fork-workflow-change merge conflict Branch has merge conflicts with its base — author must resolve before merge labels Oct 9, 2026
@amadsalmon
amadsalmon requested a review from bolichen97 October 9, 2026 18:42
@github-actions github-actions Bot added the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Oct 15, 2026
@amadsalmon
amadsalmon force-pushed the fix/gpt-review-disable-web-search branch from 430b03e to ea24873 Compare October 15, 2026 10:02
@github-actions github-actions Bot removed the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Oct 15, 2026

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me at head ea24873. I am leaving this as a comment, not an approval, because the PR edits files under .github/workflows/ and the fork workflow runs on this head still need a maintainer to approve them.

What I checked:

  • Still needed. Current main stages the codex config.toml in two places, codex-review.yml and fork-gpt-review.yml. Neither sets web_search, and the CLI is still pinned at 0.150.1.
  • Both GPT lanes get web_search = "disabled" as a top-level key. In the fork lane it sits above the first table header, so TOML does not read it as part of a table.
  • The diff touches no action pin, no permissions block, no job condition and no verdict logic. It only removes a tool the lane identity cannot call.
  • The head merges cleanly onto current main. On the merged tree, the fork GPT and web search tests in test/test_ai_review_workflows_credential_isolation.py pass.
  • Mutation check: with the codex-review.yml line removed, the new codex-review.yml case fails.

Remaining before merge:

  • A maintainer needs to approve the fork workflow runs on this head. PR Readiness is red only because those runs are waiting, and no AI lane has a verdict on this head yet.
  • The fork GPT lane runs its workflow from main. If that lane aborts on a web search denial on this PR, that is the bug this PR fixes, not the diff.

Minor, not blocking:

  • The Tests section of the description names test/test_ai_review_workflows.py, but the new class is in test/test_ai_review_workflows_credential_isolation.py. Please update that line.
  • The new class pins a codex tool setting rather than credential isolation, so a sibling module would be a more natural home. Optional.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 17, 2026
@amadsalmon

Copy link
Copy Markdown
Member Author

@bolichen97 thanks for the review. I updated the Tests line in the description to name test/test_ai_review_workflows_credential_isolation.py.

I kept the new class in that file for now. It reuses the helper there that runs each lane's "Configure the review CLI" step, and moving it would make a new head, which strips the allow-fork-workflow-change label and needs another round of fork workflow approval. If you'd still like it in a sibling module, I can move it in a follow-up PR.

Sent by an AI agent on behalf of Amad

@bolichen97

Copy link
Copy Markdown
Contributor

/ai-review override gpt ea24873: agent: the GPT lane produced no verdict because its model was refused a web-search tool it is not allowed to call, which is the exact failure this PR fixes by setting web_search to disabled; not a finding on the diff.

@github-actions

Copy link
Copy Markdown
Contributor

Human judgment recorded

@bolichen97 marked the gpt AI finding as false positive, not applicable, or explicitly accepted for ea2487391a9ff929e8f155760b8d93254a75e227.

agent: the GPT lane produced no verdict because its model was refused a web-search tool it is not allowed to call, which is the exact failure this PR fixes by setting web_search to disabled; not a finding on the diff.

This decision applies only to this commit. A new push requires a new judgment.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed allow-fork-workflow-change readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 18, 2026
@bolichen97

Copy link
Copy Markdown
Contributor

/ai-review override gpt 19783b9: agent: the GPT lane produced no verdict because its model was refused a web-search tool it is not allowed to call, which is the exact failure this PR fixes by setting web_search to disabled; not a finding on the diff.

@github-actions

Copy link
Copy Markdown
Contributor

Human judgment recorded

@bolichen97 marked the gpt AI finding as false positive, not applicable, or explicitly accepted for 19783b9233a3b6ad1122c0cebc879a44081ec3b2.

agent: the GPT lane produced no verdict because its model was refused a web-search tool it is not allowed to call, which is the exact failure this PR fixes by setting web_search to disabled; not a finding on the diff.

This decision applies only to this commit. A new push requires a new judgment.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 18, 2026
Both GPT review lanes write codex's config.toml without a web_search
key. codex 0.150.1 reads an absent key as "cached", a mode it allows
for every provider, so the reviewer is offered a web-search tool. The
lanes' Bedrock identity is not allowed to use it: when the reviewer
searches, the call is refused, the stream ends, and the required GPT
5.6 Review check fails with "review incomplete" and no verdict.

Set web_search = "disabled" at the top level of the config both lanes
write, so the tool is never offered. A test runs each lane's config
step and parses the file it writes.

Fixes kirodotdev#16142
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 18, 2026
@amadsalmon
amadsalmon force-pushed the fix/gpt-review-disable-web-search branch from 64138aa to 1253f52 Compare October 18, 2026 09:29
@amadsalmon

Copy link
Copy Markdown
Member Author

@bolichen97 thanks for updating the branch. The second "Update branch" merge took the PR to 3 commits, and PR Hygiene allows at most 2 (PR must contain 1 or 2 commits (has 3), run 39113882272). So I rebased the one commit onto main 1016b01054 as 1253f52d5a.

The diff is identical to before: the same 6 + 6 workflow lines and the same test change in 3 files. Your two merge commits carried nothing beyond main, which I checked before pushing. Locally, the 1,509 tests in test/test_ai_review_workflows*.py and every Backend Lint step pass.

The new head needs the same three steps again, sorry for the churn: the allow-fork-workflow-change label, run approval, and the GPT override. If the branch goes stale again, "Update with rebase" in the button's dropdown keeps it at one commit. Ready to paste:

/ai-review override gpt 1253f52d5a726980a8c807622d0ca2823fd3860a: agent: the GPT lane produced no verdict because its model was refused a web-search tool it is not allowed to call, which is the exact failure this PR fixes by setting web_search to disabled; not a finding on the diff.

Sent by an AI agent on behalf of Amad

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 18, 2026

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving head 1253f52.

What I checked on this head:

  • The diff is the same 3 files as the reviewed revision: codex-review.yml, fork-gpt-review.yml and the credential isolation test.
  • I read every workflow hunk line by line. Each adds web_search = "disabled" to the staged review CLI config plus a comment. No secret or token use, no permissions change, no new trigger, no remote code execution, and no review prompt text changed.
  • The new parametrized test parses the staged TOML for both GPT lanes and asserts the key is at the top level.
  • PR Readiness is passed. Every check that was cancelled by a superseded run has a successful rerun on this head, and there are no held fork runs.
  • No merge conflict, base is main.

Queuing squash auto-merge.

@bolichen97
bolichen97 merged commit 27ba1b5 into kirodotdev:main Oct 18, 2026
104 of 112 checks passed
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

allow-fork-workflow-change fork Pull request from a fork (external contributor)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ci: GPT 5.6 Review fails closed on a refused web search; the lanes leave codex web_search on

2 participants