Repository navigation
fix(ci): turn off web search in the GPT review lanes - #16161
Conversation
Design Review (Opus 5.5, fork) — ✅ PASSDesign-level review of Design-Verdict: PASS Check that a later run of the same-repo GPT 6.1 Review on this branch reaches a verdict with [DESIGN-REVIEWED] 1253f52 Description read[DESCRIPTION-READ] c2c546388155f720acaf287b06427c5bbc8d25992b2df2c4d9186f4a6984dca2 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5, fork) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Confirm that What this change shipsInventory (3 items) — 3 justifiedIntent: let the required GPT review check reach a verdict instead of failing with "review incomplete" when the reviewer tries to search the web. This is a FIX (#16142).
[FIRST-PRINCIPLES-REVIEWED] 1253f52 Description read[DESCRIPTION-READ] 64f29f053f152074e40aa68bbbb672ef37762b2ff9874e7b15ced6d44f7496bd That sha256 names the PR description this verdict read. Recompute it with |
Opus 5.5 Review (fork) — ✅ no blocking findingsReviewed |
GPT 6.1 Review (fork) — ✅ no blocking findingsReviewed Review detailsNo findings. |
|
Status for maintainers (updated in place). Current head: The two "Update branch" merges took the PR to 3 commits, over PR Hygiene's limit of 2, so the commit was rebased onto Design and First Principles both ask to confirm codex 0.150.1 accepts
So CI on Left for a maintainer: an approving review and the merge. "Update with rebase" keeps the branch at one commit if it goes stale. Sent by an AI agent on behalf of Amad |
bolichen97
left a comment
There was a problem hiding this comment.
Approving: the diff is two web_search = "disabled" lines plus a TOML-parsing test. It only removes a tool the lanes' Bedrock identity could never call. Reran the two timed-out checks.
450480f to
430b03e
Compare
430b03e to
ea24873
Compare
bolichen97
left a comment
There was a problem hiding this comment.
Looks good to me at head ea24873. I am leaving this as a comment, not an approval, because the PR edits files under .github/workflows/ and the fork workflow runs on this head still need a maintainer to approve them.
What I checked:
- Still needed. Current main stages the codex config.toml in two places, codex-review.yml and fork-gpt-review.yml. Neither sets web_search, and the CLI is still pinned at 0.150.1.
- Both GPT lanes get web_search = "disabled" as a top-level key. In the fork lane it sits above the first table header, so TOML does not read it as part of a table.
- The diff touches no action pin, no permissions block, no job condition and no verdict logic. It only removes a tool the lane identity cannot call.
- The head merges cleanly onto current main. On the merged tree, the fork GPT and web search tests in test/test_ai_review_workflows_credential_isolation.py pass.
- Mutation check: with the codex-review.yml line removed, the new codex-review.yml case fails.
Remaining before merge:
- A maintainer needs to approve the fork workflow runs on this head. PR Readiness is red only because those runs are waiting, and no AI lane has a verdict on this head yet.
- The fork GPT lane runs its workflow from main. If that lane aborts on a web search denial on this PR, that is the bug this PR fixes, not the diff.
Minor, not blocking:
- The Tests section of the description names test/test_ai_review_workflows.py, but the new class is in test/test_ai_review_workflows_credential_isolation.py. Please update that line.
- The new class pins a codex tool setting rather than credential isolation, so a sibling module would be a more natural home. Optional.
|
@bolichen97 thanks for the review. I updated the Tests line in the description to name I kept the new class in that file for now. It reuses the helper there that runs each lane's "Configure the review CLI" step, and moving it would make a new head, which strips the Sent by an AI agent on behalf of Amad |
|
/ai-review override gpt ea24873: agent: the GPT lane produced no verdict because its model was refused a web-search tool it is not allowed to call, which is the exact failure this PR fixes by setting web_search to disabled; not a finding on the diff. |
Human judgment recorded@bolichen97 marked the gpt AI finding as false positive, not applicable, or explicitly accepted for
This decision applies only to this commit. A new push requires a new judgment. |
|
/ai-review override gpt 19783b9: agent: the GPT lane produced no verdict because its model was refused a web-search tool it is not allowed to call, which is the exact failure this PR fixes by setting web_search to disabled; not a finding on the diff. |
Human judgment recorded@bolichen97 marked the gpt AI finding as false positive, not applicable, or explicitly accepted for
This decision applies only to this commit. A new push requires a new judgment. |
Both GPT review lanes write codex's config.toml without a web_search key. codex 0.150.1 reads an absent key as "cached", a mode it allows for every provider, so the reviewer is offered a web-search tool. The lanes' Bedrock identity is not allowed to use it: when the reviewer searches, the call is refused, the stream ends, and the required GPT 5.6 Review check fails with "review incomplete" and no verdict. Set web_search = "disabled" at the top level of the config both lanes write, so the tool is never offered. A test runs each lane's config step and parses the file it writes. Fixes kirodotdev#16142
64138aa to
1253f52
Compare
|
@bolichen97 thanks for updating the branch. The second "Update branch" merge took the PR to 3 commits, and PR Hygiene allows at most 2 ( The diff is identical to before: the same 6 + 6 workflow lines and the same test change in 3 files. Your two merge commits carried nothing beyond The new head needs the same three steps again, sorry for the churn: the Sent by an AI agent on behalf of Amad |
bolichen97
left a comment
There was a problem hiding this comment.
Approving head 1253f52.
What I checked on this head:
- The diff is the same 3 files as the reviewed revision: codex-review.yml, fork-gpt-review.yml and the credential isolation test.
- I read every workflow hunk line by line. Each adds web_search = "disabled" to the staged review CLI config plus a comment. No secret or token use, no permissions change, no new trigger, no remote code execution, and no review prompt text changed.
- The new parametrized test parses the staged TOML for both GPT lanes and asserts the key is at the top level.
- PR Readiness is passed. Every check that was cancelled by a superseded run has a successful rerun on this head, and there are no held fork runs.
- No merge conflict, base is main.
Queuing squash auto-merge.
Problem / Motivation
Goal: The GPT review check (GPT 6.1 Review, named GPT 5.6 Review when this was filed) reaches a verdict even when its reviewer wants to search the web, instead of failing with "review incomplete".
Both GPT review lanes write codex's
config.tomlwith noweb_searchkey, and codex 0.150.1 reads an absent key ascached. So the reviewer is offered a web-search tool that the lanes' Bedrock identity is not allowed to use. When the reviewer searches, the call is refused (Access denied: web search is not authorized for this identity), the stream ends, and the pass fails closed with no verdict. #15968 hit this on three heads in a row, and #15200 and #15967 hit it too.Why it matters
GPT 6.1 Review is a required check, so a PR whose reviewer decides to search cannot merge until a maintainer overrides the lane. A plain rerun fails the same way. It hits same-repo and fork PRs alike.
Not a goal
What changed (motivation → approach → change)
GPT 6.1 Review no longer fails when its reviewer tries to search the web. Both lanes now write
web_search = "disabled"into codex's config, so codex never offers the reviewer a search tool, and the lanes' Bedrock identity has no search call to be refused.disabledis the only mode for which codex builds no web-search tool. The key goes at the top level of the TOML incodex-review.ymlandfork-gpt-review.yml. In the fork lane it sits above the first table header,[model_providers.bedrock-signer], so TOML does not read it as part of a table.What codex does with the key, read from the source at the pinned
rust-v0.150.1tag:web_searchis a top-level config key: "disabled, cached, indexed, or live"codex-rs/config/src/config_toml.rscached:resolve_web_search_mode(&cfg, &features).unwrap_or(WebSearchMode::Cached)codex-rs/core/src/config/mod.rscachedfor every provider, so it offers the tool under Bedrock too, including the fork lane'sbedrock-signerproxy providerresolve_web_search_mode_for_turn, same filedisabledbuilds no tool:Some(WebSearchMode::Disabled) | None => return Nonecodex-rs/core/src/tools/hosted_spec.rsBackwards compatibility
Compatible: the lanes lose only a tool their identity could never use; each review still runs the same model, prompts and sandbox.
Tests
test/test_ai_review_workflows_credential_isolation.py:TestGptLanesOfferTheModelNoWebSearchruns each lane's "Configure the review CLI for Amazon Bedrock" step and parses theconfig.tomlit writes withtomllib. It fails when the key is absent, holds another mode, or lands inside a table. The helper that runs that step takes the workflow name, so the existing fork-lane config tests share it.The test catches the bug: with the workflow change reverted, both cases fail; with the key moved under
[shell_environment_policy], the fork case fails.Manual verification
N/A for a live run on this PR. The fork lane runs its workflow from
main, so this PR's own GPT 6.1 Review still uses the old config and can fail the same way. The same-repo lane runs the workflow from the PR's branch, so same-repo PRs get the fix once their branch includes it. The codex behavior above is read from the source at the version.github/review-cli/package.jsonpins.Related Issues
Fixes #16142
Pattern harvest
Rule candidate: lint
Pattern: a review lane's staged CLI config leaves a capability setting (sandbox mode, web search) to the CLI's default, so the default decides what the reviewer can do.
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)