Skip to content

ci: distinguish provider refusal from crash in the fork GPT review lane - #8895

Closed
javenciu wants to merge 1 commit into
kirodotdev:mainfrom
javenciu:fix/fork-gpt-refusal-vs-crash
Closed

javenciu wants to merge 1 commit into
kirodotdev:mainfrom
javenciu:fix/fork-gpt-refusal-vs-crash

Conversation

@javenciu

@javenciu javenciu commented Sep 6, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

When the review provider refuses to review a fork PR's diff — declines because of what the content is, as opposed to crashing or timing out — the fork GPT lane (fork-gpt-review.yml) reports the generic "review incomplete" state and advises re-running the workflow. Issue #8739 tracks porting the refusal/crash distinction that landed for the same-repo lane in #8738 (issue #8685).

The two failure modes need opposite remedies: a crash is fixed by re-running, while a refusal is caused by the diff itself and is empirically sticky — 12 consecutive identical refusals across 10 heads were measured on one PR (#8685). Fork PRs are where this matters most: an external contributor's security fix whose evidence is a working exploit is exactly the diff a provider content filter refuses, and today that contributor (and the maintainer triaging the red check) is told to re-run forever.

Why it matters

A maintainer looking at "review incomplete" on a fork PR has no way to tell a transient infrastructure failure from a permanent content refusal. The advised remedy (re-run) has never been observed to clear a refusal, so the check stays red with a misleading title, the contributor's PR looks broken rather than needing-human-review, and the one remedy that works — a maintainer reading the change directly — is never named.

What changed (motivation → approach → change)

Observed symptom: a provider refusal on a fork PR surfaces as "review incomplete" with re-run advice that does not work.

Root cause: the fork lane discards each pass's output stream, so when a pass fails (rc != 0 or empty pass file) there is nothing to distinguish the provider's refusal line from a crash — the same gap #8738 closed for codex-review.yml.

The change, a mechanical port of #8738's classification with one deliberate fork-specific difference:

  • Each review pass's combined stream is now tee'd into RUNNER_TEMP (never the checkout, so a PR cannot plant a symlink at the log's name) where the exit code is captured.
  • A failed pass is classified as refused only when the provider's own error line appears anchored at line start and only in the 4000-byte tail of that pass's captured stream. Both constraints are load-bearing here — this lane hands the model more attacker-controlled text than the same-repo lane (the prompt embeds the fork author's title/body, and the reviewer echoes the diff, which for a PR touching this workflow contains the signature verbatim) — so an echoed copy of the signature must not reclassify an ordinary crash.
  • The classification rides the verdict assembly's refused step output into both publishers (summary comment and fail-closed check-run), never a grep of the review body: on the clean path that body is model prose, and a verdict that merely quotes the refusal wording must not be reclassified.
  • The refused terminal state still fails closed — no [GPT-REVIEWED] marker, so a declined review can never read as an approval — but the check-run title says "reviewer refused this diff" and the comment names the honest remedy.
  • Fork-specific difference, and why this is not a blind copy: the same-repo lane's refused copy routes to /ai-review override. This lane resolves no override record, so promising that command here would route a maintainer to a clearance path that cannot clear this check. The refused copy routes to direct maintainer review instead.
  • A refused run rides the shared guarded_comment_upsert withhold path (fork review lanes: incomplete review body still overwrites a posted verdict (fork-lane twins of #8292) #8344, fix(ci): guard the review lanes against verdict overwrite (#8344) #8450): the refused body carries no [GPT-REVIEWED] proof marker, so it never modifies an existing comment and only creates the first placeholder when a successful lookup confirms none exists. This PR pins that behavior for the refused kind with tests rather than growing a posting branch of its own.
  • A refusal dominates a mixed failure (one pass crashed, the other refused), because a re-run only helps if the refusal does not recur.

Alternatives considered: grepping the review body for a refusal marker downstream (rejected — the body on the clean path is model prose, and a marker nothing may parse would invite prose-grepping); matching the signature anywhere in the whole stream (rejected — an echoed copy in PR-controlled text would reclassify crashes); porting the same-repo lane's /ai-review override remedy text verbatim (rejected — this lane cannot resolve an override record, see above).

The claude-review.yml half of #8739 is deliberately not included: that lane's output is action-mediated, so refusal classification needs its own signature source rather than this mechanical port. See Pattern harvest.

docs/ci/ci-and-reviews.md (the owning doc for the reviewer lanes) is updated in the same commit: the refusal paragraph now covers both GPT lanes and names each lane's remedy.

Tests

New class TestForkGptRefusalTerminalState (12 tests) in test/test_ai_review_workflows.py, mirroring the same-repo lane's coverage plus the fork-specific attack surface:

  • test_refusal_is_classified_where_rc_is_captured — both passes tee into RUNNER_TEMP, match line-anchored against the tail, and record into the re-initialized refused-passes file; the signature carries the provider's ERROR: prefix and stays free of regex metacharacters (it is interpolated into an anchored grep pattern).
  • test_signature_stays_in_sync_with_the_same_repo_lane — both lanes classify against one identical signature; drift would silently split refusal semantics between lanes.
  • test_provider_emitted_refusal_line_classifies_as_refused — executes the workflow's real classification block (extracted from the YAML, run under bash) against a fabricated stream carrying the provider's line: classified refused.
  • test_echoed_signature_in_pr_controlled_text_stays_a_crash — the signature quoted/indented in PR-controlled text (exactly what this PR's own diff looks like to the reviewer) does not classify: a crash on such a PR stays re-runnable.
  • test_signature_outside_the_stream_tail_stays_a_crash — an early echoed copy scrolled past by 8000 bytes of output does not classify.
  • test_refused_pass_publishes_its_own_terminal_state — refused body carries no [GPT-REVIEWED]/[GPT-REFUSED] marker, no re-run advice, no /ai-review override, routes to direct maintainer review, and emits refused=true to GITHUB_OUTPUT.
  • test_crashed_pass_keeps_the_rerunnable_incomplete_state — the pre-existing incomplete state and its re-run advice are untouched for genuine crashes.
  • test_refusal_dominates_a_mixed_failure — one crashed + one refused ⇒ refused wins.
  • test_clean_run_still_publishes_pass_2_verbatim — the happy path is byte-identical.
  • test_refused_run_never_modifies_an_existing_comment — the GPT review verdict lives in one mutable comment, so an incomplete run hides a blocking finding #8292 stale-read guard covers the refused kind (no PATCH, no create with an existing comment).
  • test_refused_run_with_no_existing_comment_creates_the_refused_placeholder — first comment carries the refused verdict and fork-honest remedy.
  • test_refused_classification_rides_step_outputs_into_both_publishers — comment and finalize steps read steps.gpt_pass2.outputs.refused through env, never a body grep; the check-run title changes but its failure conclusion does not.

Fails-before, run at base d4c2cbf22 with the new tests committed onto the pristine tree (full targeted file, tests only):

11 failed, 443 passed
(the new class alone: 11 failed, 1 passed — the 1 pass is the
 stale-read-guard control, which holds via the shared
 guarded_comment_upsert withhold path (#8344, #8450) — correct
 control behavior)

With the fix: 12 passed; full targeted file: 454 passed.

Manual verification

N/A — the workflow's classification and publication logic is executed directly by the tests (the classification block and verdict assembly are extracted from the YAML and run under bash with fabricated streams), which is the same technique the file's existing suites use for this lane. A live refusal requires the provider to refuse a real diff on a real fork PR, which cannot be staged deterministically.

Related Issues

Fixes #8739 (the fork-gpt-review.yml half; the claude-review.yml half needs its own signature source — see Pattern harvest). Ports #8738 (#8685). Interacts with the shared guarded_comment_upsert withhold path (#8344, #8450), which this pins for the refused kind with tests.

Pattern harvest

Rule candidate: review-prompt
Pattern: "failure-state classification must ride step outputs, never a grep of content that is model prose or attacker-influenced on other paths"

The remaining half of #8739 — claude-review.yml's action-mediated lane, where the provider's stderr is not directly capturable and refusal classification needs its own signature source — is a distinct follow-up, not covered by this port.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Per the template, the CLA wording is a placeholder pending OSPO/Legal; nothing to affirm beyond the checklist until it is supplied.

@javenciu
javenciu requested review from a team and cixuuz September 6, 2026 05:35
@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: action required A blocking check or review needs attention merge conflict Branch has merge conflicts with its base — author must resolve before merge labels Sep 6, 2026
@javenciu
javenciu force-pushed the fix/fork-gpt-refusal-vs-crash branch from 633808c to a0765b8 Compare September 6, 2026 10:14
@github-actions github-actions Bot added merge conflict Branch has merge conflicts with its base — author must resolve before merge and removed merge conflict Branch has merge conflicts with its base — author must resolve before merge labels Sep 6, 2026
@bolichen97
bolichen97 force-pushed the fix/fork-gpt-refusal-vs-crash branch from a0765b8 to 4e3aab1 Compare September 8, 2026 15:33
@bolichen97

Copy link
Copy Markdown
Contributor

Rebased onto main 9c2b4b678 by a maintainer as part of the 2026-09-08 open-PR audit (was 172 commits behind, mergeable_state: dirty).

Conflicts and resolution:

  • test/test_ai_review_workflows.py: this PR appends TestForkGptRefusalTerminalState at the end of the file, and main appended new classes there too. Took main's file and re-appended this PR's 314 lines unchanged, so both sets of classes are present.
  • .github/workflows/fork-gpt-review.yml and docs/ci/ci-and-reviews.md auto-merged; main's Opus 4.8 / model-name churn is preserved alongside this PR's refusal handling.

Gates run locally on the changed file: black, isort, flake8 clean; pytest test/test_ai_review_workflows.py 536 passed.

Please review the resolution. A maintainer push makes the maintainer the last pusher, so under the repo's last-push rule a second approver is needed. Reply if anything looks wrong.

@github-actions github-actions Bot added merge conflict Branch has merge conflicts with its base — author must resolve before merge and removed merge conflict Branch has merge conflicts with its base — author must resolve before merge labels Sep 8, 2026
@NicholasRBowers NicholasRBowers added the needs-pr-triage PR scanner: awaiting automated triage label Sep 11, 2026
@chenmingwei23 chenmingwei23 added needs-author-decision PR blocked on author input and removed needs-pr-triage PR scanner: awaiting automated triage labels Sep 11, 2026
@chenmingwei23

Copy link
Copy Markdown
Contributor

Kiro Crew [operator: chenmingwei23#de330d0c]: This PR has been inactive for 7+ days. I reviewed the blockers but they require a maintainer's input:

  • This PR changes a CI workflow file (the fork GPT review workflow), so its four workflow runs sit in awaiting maintainer approval and none has run yet. The pipeline does not auto-approve fork PRs that touch workflow files; a maintainer must approve the runs before CI can even evaluate the change.
  • A maintainer already rebased this onto main and noted that because the maintainer was the last pusher, the repo last-push rule needs a second approver to merge.

There is no mechanical CI failure to fix here; the gate is human approval. When a maintainer approves the workflow runs and provides the second review, the pipeline will re-assess on its next cycle.

@iamwhatever
iamwhatever force-pushed the fix/fork-gpt-refusal-vs-crash branch from 4e3aab1 to d31b0c8 Compare September 21, 2026 21:23
@iamwhatever

Copy link
Copy Markdown
Contributor

Rebased onto current origin/main by a maintainer (maintainer_can_modify), force-pushed once to fix/fork-gpt-refusal-vs-crash. New head d31b0c89f.

How the change was re-applied. Two conflicts, both resolved by union rather than by restoring this branch's older copy of either file:

  • .github/workflows/fork-gpt-review.yml — the "Finalize check-run (fail closed)" env: block. Main had added WR_RUN_ID / WR_RUN_ATTEMPT; this branch adds REFUSED. All three are now present. Everything else in the file auto-merged onto main's current shape, including the harden-runner egress allowlist that landed since this branch was written.
  • test/test_ai_review_workflows.py — an append-at-EOF collision. Main's new classes are kept in full and TestForkGptRefusalTerminalState is appended after them.

The refusal/crash distinction itself is unchanged in behaviour: signature env var on both pass steps, 2>&1 | tee into RUNNER_TEMP, line-anchored tail-scoped classification, the refused step output, and the refused branches in the comment and check-run finalizers.

No security boundary was widened. Concretely:

  • permissions: is byte-identical to main — actions: read, checks: write, contents: read, id-token: write, pull-requests: write.
  • The harden-runner step, its egress-policy: block, and its whole allowed-endpoints list are byte-identical to main (diff over that block is empty). No host added.
  • No secret added, renamed or referenced.
  • Triggers unchanged: workflow_run on Fast Gate, types completed.
  • No fork-PR guard relaxed.

The whole 3-dot diff removes exactly 9 lines from the workflow, and every one is a line this change means to replace (the two bare "$PROMPT" || rc=$? invocations, the two generic "did not complete" warnings, and the 5-line incomplete-review body that the refusal branch now wraps). Totals are 3 files, +447/-21, matching the pre-rebase shape.

Gates run locally against this head, using the repo's own scripts:

  • test/test_ai_review_workflows.py — 794 passed (this is the file that asserts on the workflow YAML, so it is the real gate here).
  • scripts/docs_lint.py --test and scripts/docs_lint.py — pass.
  • scripts/check_comment_history.py scoped to origin/main — pass. It initially failed on three issue-number references inside this change's own added docstring/comment lines, so those references were dropped; wording is otherwise unchanged and no assertion depends on them.
  • check_brand_name, check_focus_cue, check_feature_map, check_changelog_history, check_builtin_skill_scope, check_loop_bound_locks, check_testpaths_coverage, check_harness_parity, check_memory_store_seam — all pass.
  • black --check, flake8, isort --check-only on the touched test file — clean.
  • No actionlint or zizmor job exists in .github/workflows/; both names appear only in comments.

The fork workflow-change guard is expected to flag this PR for human attention, since the change edits a workflow file from a fork. That is the guard working as intended and was not suppressed.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed merge conflict Branch has merge conflicts with its base — author must resolve before merge readiness: action required A blocking check or review needs attention labels Sep 21, 2026
@github-actions

github-actions Bot commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review (fork) — ✅ no blocking findings

Reviewed 0d8eaa4a03afdc0b7430d43bf5a6b80e163ad65f via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 0d8eaa4

@github-actions

github-actions Bot commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5 Review (fork) — ✅ no blocking findings

Reviewed 0d8eaa4a03afdc0b7430d43bf5a6b80e163ad65f via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 0d8eaa4

@github-actions

github-actions Bot commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5, fork) — ✅ PASS

Design-level review of 0d8eaa4a03afdc0b7430d43bf5a6b80e163ad65f via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

On the first real fork-lane refusal, confirm the check title reads "reviewer refused this diff" — live refusals were unstageable here.

[DESIGN-REVIEWED] 0d8eaa4

@github-actions

github-actions Bot commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5, fork) — ✅ PASS

Premise-level review of 0d8eaa4a03afdc0b7430d43bf5a6b80e163ad65f via the fork AI-review pipeline — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

Confirm the hardcoded REFUSAL_SIGNATURE still matches the provider CLI's current wording — drift silently reverts refusals to futile re-run advice in both lanes.

Verified against the base: codex-review.yml already ships this exact classification (signature, tail-scoped anchored grep, refused step output), so the port claim is real; the base fork-gpt-review.yml has none of it, and the base doc itself recorded the gap ("still reports only the generic incomplete state and is tracked separately") — the defect's provenance is a documented, tracked gap plus linked issues (#8739, #8685). The one fork-specific deviation rests on a claim I checked: ai-review-human-override.yml handles fork PRs for the gpt target only by re-running fork-gpt-review.yml (line 310), which cannot clear a sticky refusal, so routing the refused copy to direct maintainer review instead of the override command is honest. Only two lanes invoke the codex CLI (grep review-cli/node_modules/.bin/codex: 2 files); the other already has the fix, and the action-mediated claude-review.yml half is declared deferred with a mechanism-level reason. The refused output has 2 counted consumers. No undeclared items.

What this change ships

Inventory (8 items) — 8 justified

Intent: stop telling fork contributors and maintainers to re-run a review the provider permanently refuses, by naming the refusal as its own failing terminal state — a FIX.

  1. Refused fork review's required check title now reads "reviewer refused this diff", still failing — justified
  2. Refused run's PR comment names direct maintainer review and drops re-run advice — justified
  3. A refusal outranks a crash when both passes fail — justified
  4. Each pass's output stream is now captured into RUNNER_TEMP logs — justified
  5. New refused step output carries the classification into both publishers — justified
  6. Genuine crashes keep the byte-identical re-run state — justified
  7. The owning CI doc's refusal paragraph now covers both GPT lanes — justified
  8. 12 tests pin the classification, spoof resistance, and the guarded upsert path — justified

[FIRST-PRINCIPLES-REVIEWED] 0d8eaa4

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision merge conflict Branch has merge conflicts with its base — author must resolve before merge and removed readiness: checking Automated validation is still running labels Sep 22, 2026
@iamwhatever iamwhatever self-assigned this Sep 23, 2026
A provider refusal and a crash both arrive as a nonzero exit or an empty
pass file, but they mean opposite things: a crash is fixed by re-running,
while a refusal is caused by what the diff is and recurs identically
across re-runs. The same-repo lane already classifies the two (kirodotdev#8685,
via kirodotdev#8738); the fork lane -- where an untrusted working exploit is most
likely to arrive -- still reported every refusal as a re-runnable
"review incomplete".

Port the classification: tee each pass's combined stream into
RUNNER_TEMP where the exit code is captured, match the provider's own
refusal line anchored at line start and only in the 4000-byte tail
(the stream carries PR-controlled text -- the prompt embeds the fork
author's title/body and the reviewer echoes the diff, so an echoed copy
of the signature must not reclassify an ordinary crash), and ride the
result through the verdict assembly's refused step output into both
publishers, never a body grep. The check still fails closed; its title
and the summary comment now name the refusal and route to direct
maintainer review instead of /ai-review override, which this lane
cannot resolve, or re-run advice that has never been observed to work.

The claude-review.yml half of kirodotdev#8739 is deliberately not ported here:
that lane's output is action-mediated, so refusal classification needs
its own signature source rather than this mechanical port.

Fixes kirodotdev#8739 (fork-gpt-review.yml half)
@iamwhatever
iamwhatever force-pushed the fix/fork-gpt-refusal-vs-crash branch from d31b0c8 to 0d8eaa4 Compare September 23, 2026 05:11
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Sep 23, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor

Rebased onto main to resolve test/test_ai_review_workflows.py; no code change. Main's new classes in that file are kept and this PR's TestForkGptRefusalTerminalState is appended after them; 800 tests in the file pass. Diff shape is unchanged at 3 files +447/-21. The fork-gpt-review.yml diff still touches no allowed-endpoints, no permissions:, no uses: line and no checkout step (the single "checkout" word in the diff is inside an added comment).

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running and removed merge conflict Branch has merge conflicts with its base — author must resolve before merge readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 23, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor

Red lanes on 0d8eaa4 are not this PR's: Backend Tests (3.12, 7) and (Windows) (7) both fail test_crew_log_exc_info_sites.py::test_the_exc_info_sites_left_in_crew_log_are_exactly_the_vetted_ones on the new emit.py:_record_session_tree_edge and session_tree_projection.py sites main added in #12757, which reproduces on a clean origin/main checkout (b83100d) with no PR content; Coverage Gate fails closed on those shards; and Frontend Lint & Type Check fails the restyle ratchet at 347/50 vs baseline 349/51 on website/src/pages/members/MembersPage.tsx, also reproduced on plain main. This PR touches only .github/workflows/fork-gpt-review.yml, docs/ci/ci-and-reviews.md and test/test_ai_review_workflows.py. Fork workflow-change guard is blocked pending a maintainer override label.

@github-actions github-actions Bot added the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Sep 27, 2026

Copy link
Copy Markdown
Contributor

PR triage (2026-10-05): DISTINCT, needs a REBASE. Overlaps with #12529 and #16161, which should be discussed before merging.

I checked the diff at head 0d8eaa4a0 against origin/main af1eb612d.

Not covered on main. On main, only codex-review.yml has the refusal/crash classification. It came in with merged PR #8738 (fixing issue #8685) and uses REFUSAL_SIGNATURE, the tail-scoped anchored grep, and the refused output. Main's fork-gpt-review.yml has none of it: both passes still run "$PROMPT" || rc=$? with no tee, and the finalizer title is still only review incomplete. docs/ci/ci-and-reviews.md on main still says the fork lane "is tracked separately". Issue #8739 is still open. This PR is the only open change for the fork-GPT half of that issue. I found no breaking change and no P0 bug: the refused state still fails closed, and nothing on main parses the check-run title.

Rebase needed. mergeable_state: dirty. The merge base is 1784 commits behind main, and git merge-tree conflicts in .github/workflows/fork-gpt-review.yml and test/test_ai_review_workflows.py. Things to fix during the rebase:

  • Main renamed the steps to GPT 6.1 review (discovery pass) / (falsification pass), and the log copy now says GPT 6.1. The new tests look the steps up by the name GPT 5.6 review (discovery pass), and the added ::warning:: lines and the comment copy still say GPT 5.6 / "Fork GPT 5.6 Review".
  • The inline comment points to (see "the fork lane has no override" below), but that phrase doesn't appear anywhere in the file.
  • Since ci: let readiness read human overrides for superseded fork blocks #16755, readiness accepts /ai-review override records for fork lanes when a superseded BLOCK is cleared. Please check that the refused copy ("direct maintainer review, no override command") is still the right remedy for a failed, unstamped fork check. The override arm still only re-runs fork-gpt-review.yml (ai-review-human-override.yml:311).

Merge discussion (OVERLAPPING, complementary):

Recommendation: keep it open and rebase. Don't close it.

@javenciu javenciu closed this Oct 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor) merge conflict Branch has merge conflicts with its base — author must resolve before merge needs-author-decision PR blocked on author input readiness: action required A blocking check or review needs attention

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ci: fork GPT and Opus review lanes still cannot tell a reviewer refusal from a crash

5 participants