Skip to content

fix(ci): stop the fork advisory verdict captures aborting their own step - #15845

Merged
iamwhatever merged 1 commit into
mainfrom
replace/12517-fork-verdict-capture
Oct 1, 2026
Merged

iamwhatever merged 1 commit into
mainfrom
replace/12517-fork-verdict-capture

Conversation

@iamwhatever

Copy link
Copy Markdown
Contributor

Replaces #12517 by @leonlaiyc (same change, moved in-repo because the fork workflow-change guard blocks all bots on a fork PR under .github/**, and the fork lane does not consume an override). Original authorship preserved on the commit; co-authored by the original author.

Problem / Motivation

Goal: On a fork PR, make the Design Review and First Principles lanes always post their verdict (or a readable UNKNOWN), instead of silently posting nothing when the review has no header or is very long.

On a fork PR, the Design Review and First Principles lanes can finish their
model call, hold a perfectly good verdict, and post nothing at all — no comment,
no verdict output, and nothing in the job log naming why.

Both lanes capture their verdict with

v="$(printf '%s\n' "$summary" | grep -iE '^Design-Verdict:' | head -n1 | sed -E '…' | tr … )"
[ -n "$v" ] && verdict="$v"

inside a run: block that opens set -uo pipefail and, having no shell: key,
is executed by the runner as bash -e. Two ordinary model outcomes make that
assignment's status non-zero, and -e then kills the step one line above the
fallback written for exactly those outcomes
:

  • no header — grep matches nothing and exits 1; pipefail promotes that to
    the pipeline's status.
  • a long review — head -n1 exits after the first line and closes the pipe,
    so its producer dies of SIGPIPE, status 141.

Why it matters

The verdict="UNKNOWN" default and the [ -n "$v" ] test directly beneath the
capture are a complete, fail-closed answer for a review that names no verdict.
Neither line runs. The outcome is not "UNKNOWN" but nothing: the lane writes no
verdict output, its status step reads an empty verdict, and the comment that
would have told a contributor what happened is never posted. A red either way,
but only one of them says why — and on a fork PR the contributor cannot re-run
the lane themselves.

The 141 arm is the worse one: a verdict is already in hand when the read ends
early, so an expression that merely suppressed the status would hand the lane a
verdict read out of a document the step never finished reading.

Not a goal

  • The anchor is not widened. It stays ^…-Verdict: with no leading-whitespace
    class, so exactly the same set of headers counts as before. Accepting an
    indented header is a separate question with its own contract argument (fix(ci): scope review must not abort on a headerless verdict #12351
    settled it for the scope lanes); answering it here would mix a behaviour change
    into an abort fix.
  • The same-repo design-review.yml and first-principles-review.yml are not
    touched.
    Their post-summary steps carry no set - line, so an unmatched
    grep's status is masked by the pipeline's last command and the step cannot die
    this way. Converting them would be a change with no defect behind it. (The
    fork-ux-review.yml / ux-review.yml captures already guard the grep arm with
    { … || true; }; out of scope here, kept to what fix(ci): stop the fork advisory verdict captures aborting their own step #12517 touched.)
  • No workflow trigger, permission, job, step name, check-run name or output is
    added, renamed or removed.
    Only two expressions change, one inside one
    existing step each.

What changed (motivation → approach → change)

Symptom → a fork advisory lane that reviewed fine posts no verdict and no
explanation.

Root cause → the capture is a pipeline in a step that turns a pipeline's
inner failure into the step's failure. Each of these steps already uses a
here-string twice — trimming the summary to the header, and checking for the
[…-REVIEWED] marker — each time under a comment saying a long summary must not
manufacture a SIGPIPE status under pipefail. The capture sitting between them
is the one place that rule was not applied.

Change → both captures become the one-awk-pass form #12351 introduced:

v="$(awk '!found && tolower($0) ~ /^design-verdict:/ { sub(/^[^:]*:[ \t\v\f\r]*/, ""); gsub(/\r/, ""); value = toupper($0); found = 1 } END { if (found) print value }' <<< "$summary")"

awk exits 0 whether or not the header is there, so the [ -n ] test below
decides — as the code was always written to. There is no exit on first match:
the whole summary is read and the FIRST header is printed from END, so a
verdict is never taken from a partial read. The character class spells out the
five characters [[:space:]] matches inside a record, because the runners' awk
is mawk.

Two existing pins moved as part of the diff: TestFirstPrinciplesReview and
TestTheOutputDietTightenedAndKeptItsMachineReadLines asserted the literal
grep -iE '^First-Principles-Verdict:', and _capture_step used the same
literal as a slice boundary. All three now key on an anchored reference to the
header
rather than on the program that reads it — the property those pins were
about — because a pin keyed to one tool stops finding the capture the day the
capture changes tool and then measures nothing.

Backwards compatibility

Compatible. The capture returns the same value for every input that produced a
value before — the three verdict-carrying cases (plain, lowercase, crlf)
pass on pristine origin/main as well as on this branch, so they measure the old
expression and the new one to be equal. The only inputs whose behaviour changes
are the two that produced no value and killed the step; they now produce no value
and let the step continue to its own fallback. No workflow trigger, permission,
job, step name, check-run name or output changes.

Tests

test/test_ai_review_workflows.py —
TestForkVerdictCaptureNeverAbortsAboveItsOwnFallback, 14 tests over the two
lanes. Each test extracts the step's own expression from the yaml and runs it
under bash -e with the step's own set -uo pipefail, then asserts both the
exit status and the captured value.

Six review kinds per lane:

kind expected what it pins
no-header "" grep's exit 1 no longer takes the step down
header-shaped-prose "" the anchor still tells a header from a mention of one
many-headers (60 000) PASS SIGPIPE 141 gone; first header still wins
plain CONCERNS ordinary read unchanged
lowercase BLOCK the value is still upper-cased
crlf BLOCK no carriage return left inside the value

Plus test_the_step_still_carries_the_fallback_this_pin_is_about, which fails if
verdict="UNKNOWN" or the [ -n "$v" ] test is ever removed.

Local run (this box, in-repo branch off fresh origin/main): whole file
test/test_ai_review_workflows.py — 1123 passed; the new class alone — 14
passed
. Both edited workflow files re-parsed with yaml.safe_load.

Manual verification

The mechanism was reproduced in #12517 directly, running each lane's own capture
line under bash -e with set -uo pipefail: no-header → rc 1, 60 000 headers →
rc 141, both one line above the [ -n "$v" ] fallback. workflow_run lanes
execute the default branch's yaml, so this change is unexercised until after
merge; the first fork PR opened afterwards is the real end-to-end check.

Related Issues

None. This is the counted, unconverted half of #12351's First Principles review,
which named both fork-design-review.yml and fork-first-principles-review.yml
as the siblings that abort one line above their own [ -n "$v" ] fallbacks.

Pattern harvest

Rule candidate: review-prompt

Pattern: a step's preamble (set -euo pipefail) makes a command's exit status
load-bearing, so an expression written to return "nothing found" instead kills
the step above the branch that handles "nothing found".

The prompt this proposes: when a fix converts a pipeline for exit-status reasons,
grep the sibling lanes for the same expression before calling the defect closed —
and when reviewing any run: block that opens with pipefail, ask of every
$( … | … ) whether its "found nothing" status is a value the next line reads or
a signal that kills the step first.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@iamwhatever
iamwhatever requested a review from a team as a code owner October 1, 2026 12:23
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 1, 2026
@github-actions

github-actions Bot commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5) — ✅ PASS

Premise-level review of 8cac21f1f2093680f550b7c5557c325413cb9e7f — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

Deferred sibling to confirm: fork-ux-review.yml:1279 runs its capture under the same pipefail (line 1135) with only the grep arm || true-guarded.

What this change ships

Inventory (5 items) — 5 justified

Intent: FIX — make the two fork advisory lanes post their verdict (or a readable UNKNOWN) instead of dying one line above their own fallback.

  1. Fork Design Review: a header-less or very long review now reaches the UNKNOWN fallback and posts — justified
  2. Fork First Principles: same two abort arms removed, same fallback now reached — justified
  3. A long review's verdict is now read from the whole document, never a partial read — justified
  4. Three test pins now accept either spelling of the anchored header read — justified
  5. 14 new tests run each lane's own extracted expression under the step's real preamble — justified

Verified against the checkout: both fork steps open set -uo pipefail in a run: block with no shell: key (fork-design-review.yml:730, fork-first-principles-review.yml); the same-repo post-summary steps (design-review.yml:690–1635, first-principles-review.yml:358–1382) carry no set - line, so their surviving grep|head pipelines (design-review.yml:820, first-principles-review.yml:568) cannot abort this way — the "Not a goal" claims hold. The awk form reuses the existing security-scope spelling (security-scope-review.yml:1709), not a second invention. Pin rewrites are declared and keep the anchored-header property, including rejecting the trim as a false match. Remaining pipeline captures counted (pattern -Verdict:' | head): 4, of which only fork-ux-review.yml:1279 sits under pipefail; its producer arm is unguarded but the description defers it explicitly, and the frozen Goal names only these two lanes.

[FIRST-PRINCIPLES-REVIEWED] 8cac21f

Description read

[DESCRIPTION-READ] eec12d2e9341b1f4adb99e547709271256062d8bd4cb4b603c9a6ed803c07fd2

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ✅ no blocking findings

GPT 5.6 completed its review of 8cac21f1f2093680f550b7c5557c325413cb9e7f and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 8cac21f

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 8cac21f1f2093680f550b7c5557c325413cb9e7f: <one-sentence reason>

@github-actions

github-actions Bot commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — ✅ PASS

Design-level review of 8cac21f1f2093680f550b7c5557c325413cb9e7f — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Verify the first fork PR after merge posts a verdict end-to-end — workflow_run lanes run main's yaml, so CI here never exercises the fix.

[DESIGN-REVIEWED] 8cac21f

Description read

[DESCRIPTION-READ] cc4fb4e9646fa013bf674b4d6069cf535ab06ee9a6e64209ae501fa6047b2952

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5 Review — ✅ no blocking findings

Reviewed 8cac21f1f2093680f550b7c5557c325413cb9e7f — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 8cac21f

Verdict parsed from the review's SHA-scoped output markers for commit 8cac21f1f2093680f550b7c5557c325413cb9e7f.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 8cac21f1f2093680f550b7c5557c325413cb9e7f: <one-sentence reason>

the two Security Scope lanes. Its two counted siblings were left: the fork
Design Review and fork First Principles lanes read their header the old
way, inside a step that carries `set -uo pipefail`.

Both statuses that shape produces abort the step under the runner's own
`bash -e`, one line ABOVE the `[ -n "$v" ] && verdict="$v"` fallback
written for exactly those cases. A review with no header makes grep exit 1
and pipefail promotes it; a long review makes `head -n1` close the pipe and
its producer die of SIGPIPE with 141. The lane then writes no verdict
output at all, and the comment that would have named the cause is never
posted. Reproduced locally on both lanes: rc=1 with a headerless body,
rc=141 at 60k headers, and the fallback unreached in all four runs.

Each of these steps already uses a here-string twice, each time under a
comment saying a long summary must not manufacture a SIGPIPE status under
pipefail. The capture between them was the one place the rule was not
applied, which is what makes this an oversight rather than a judgement.

Convert both to the same awk expression #12351 used, with the anchor left
exactly as it was, so which headers count does not change. The same-repo
design and first-principles lanes are deliberately untouched: their steps
carry no `set -` line, so an unmatched grep's status is masked by the
pipeline's last command and they cannot die this way.

Two existing pins named `grep` as the program that reads the header. They
now assert an ANCHORED reference to the header instead, because a pin keyed
to one tool stops finding the capture the day the capture changes tool and
then passes while measuring nothing. `_capture_step`'s slice boundary takes
both spellings for the same reason.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
(cherry picked from commit 93229f9)
@iamwhatever
iamwhatever force-pushed the replace/12517-fork-verdict-capture branch from 16cfee9 to 8cac21f Compare October 1, 2026 12:43
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 1, 2026

@buluoray buluoray left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve at 8cac21f1f2093680f550b7c5557c325413cb9e7f.

The change is two expressions, one per lane: .github/workflows/fork-design-review.yml:768 and .github/workflows/fork-first-principles-review.yml:576. Each grep | head -n1 | sed | tr verdict capture becomes the single-pass awk … <<< "$summary". I checked it against source at this head.

Security (fork / workflow_run)

  • No trigger, permissions:, job, if:, env: or shell: changes. Both lanes are still workflow_run-only, run from the default branch, with the same top-level permissions. Nothing here touches pull_request_target.

  • No new ${{ … }} interpolation in run:. The model output is still read from $summary through a here-string, and the awk program is a fixed single-quoted literal.

  • No output injection: awk prints one record with every \r removed, so verdict cannot carry a newline into $GITHUB_OUTPUT. That is the same as before.

  • Not fail-open. For every input that produced a value before, the new expression returns the same value: first anchored header, leading separator stripped, upper-cased. The tests check this on plain, lowercase and crlf. The only inputs that behave differently are the two that used to abort the step:

    • No header: this now lands on verdict="UNKNOWN", which the finalize step maps to neutral "review incomplete". An empty output maps there too.
    • Long review: the first header is now read from a complete read. The FP marker check [FIRST-PRINCIPLES-REVIEWED] $HEAD_SHA still runs after the capture, and so does the Design [DESIGN-REVIEWED] gate.

    The long-review fix also closes a small fail-open in the old code. A long review carrying BLOCK used to abort the step, so the post step (if: always()) published the generic notice instead of the …-Verdict: BLOCK line that readiness reads. That BLOCK was dropped. It now posts.

  • mawk: the [ \t\v\f\r] class in sub() is the same one already running on ubuntu-latest in fork-security-scope-review.yml:660 and security-scope-review.yml:1709.

Mutation checks (in a local worktree, then restored)

  • I put back the old grep | head capture in both lanes. 6 of 14 TestForkVerdictCaptureNeverAbortsAboveItsOwnFallback::test_every_ordinary_review_reaches_the_fallback_with_the_right_value cases failed: no-header, header-shaped-prose and many-headers, ×2 lanes.
  • I dropped the ^ anchor in the Design awk. Two tests failed: [header-shaped-prose-fork-design-review.yml-…] and TestReviewLanesPublishOnlyTheReview::test_every_lane_trims_to_its_own_header.
  • Pristine head: 58/58 pass across the new class and the three re-pinned classes.

Bot lanes: FP PASS, Design PASS, GPT and Opus no findings; 86 check-runs green, 8 skipped.

  • FP's deferred sibling is real but outside this PR. fork-ux-review.yml:1279 guards only the grep arm (|| true), and its head -n1 can still SIGPIPE under pipefail. It needs a follow-up.
  • Design's "verify the first fork PR after merge" is correct: workflow_run runs main's yaml, so these tests are the only check before merge.

Non-blocking. Please fix before merge, because these land in main's history:

  1. The squash commit body begins mid-sentence ("the two Security Scope lanes. Its two counted siblings…"). The original first line, #12351 replaced a grep | head -n1 verdict capture with one awk pass in, began with #, and the cherry-pick dropped it as a comment. Please restore it in the squash message.

  2. The PR body and commit message overstate the symptom. "posts nothing at all — no comment" and "the comment that would have named the cause is never posted" are not accurate. The post steps are if: always() (fork-first-principles-review.yml:630, fork-design-review.yml:811), so an aborted capture still posted the generic "no parseable verdict" notice, and the finalize step still published neutral "review incomplete". The real effects were:

    • a red job for a headerless review, with the same comment and check result as now;
    • a lost verdict, BLOCK included, for a review over the pipe buffer.

    Neither affects the fix itself.

@iamwhatever
iamwhatever enabled auto-merge (squash) October 1, 2026 15:56
@iamwhatever
iamwhatever merged commit a4c73e1 into main Oct 1, 2026
187 of 189 checks passed
@iamwhatever
iamwhatever deleted the replace/12517-fork-verdict-capture branch October 1, 2026 15:57
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants