Repository navigation
fix(ci): stop the fork advisory verdict captures aborting their own step - #15845
Conversation
First Principles Review (Fable 5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Deferred sibling to confirm: What this change shipsInventory (5 items) — 5 justifiedIntent: FIX — make the two fork advisory lanes post their verdict (or a readable UNKNOWN) instead of dying one line above their own fallback.
Verified against the checkout: both fork steps open [FIRST-PRINCIPLES-REVIEWED] 8cac21f Description read[DESCRIPTION-READ] eec12d2e9341b1f4adb99e547709271256062d8bd4cb4b603c9a6ed803c07fd2 That sha256 names the PR description this verdict read. Recompute it with |
GPT 5.6 Review — ✅ no blocking findingsGPT 5.6 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
Design Review (Fable 5) — ✅ PASSDesign-level review of Design-Verdict: PASS Verify the first fork PR after merge posts a verdict end-to-end — [DESIGN-REVIEWED] 8cac21f Description read[DESCRIPTION-READ] cc4fb4e9646fa013bf674b4d6069cf535ab06ee9a6e64209ae501fa6047b2952 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
Opus 5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
the two Security Scope lanes. Its two counted siblings were left: the fork Design Review and fork First Principles lanes read their header the old way, inside a step that carries `set -uo pipefail`. Both statuses that shape produces abort the step under the runner's own `bash -e`, one line ABOVE the `[ -n "$v" ] && verdict="$v"` fallback written for exactly those cases. A review with no header makes grep exit 1 and pipefail promotes it; a long review makes `head -n1` close the pipe and its producer die of SIGPIPE with 141. The lane then writes no verdict output at all, and the comment that would have named the cause is never posted. Reproduced locally on both lanes: rc=1 with a headerless body, rc=141 at 60k headers, and the fallback unreached in all four runs. Each of these steps already uses a here-string twice, each time under a comment saying a long summary must not manufacture a SIGPIPE status under pipefail. The capture between them was the one place the rule was not applied, which is what makes this an oversight rather than a judgement. Convert both to the same awk expression #12351 used, with the anchor left exactly as it was, so which headers count does not change. The same-repo design and first-principles lanes are deliberately untouched: their steps carry no `set -` line, so an unmatched grep's status is masked by the pipeline's last command and they cannot die this way. Two existing pins named `grep` as the program that reads the header. They now assert an ANCHORED reference to the header instead, because a pin keyed to one tool stops finding the capture the day the capture changes tool and then passes while measuring nothing. `_capture_step`'s slice boundary takes both spellings for the same reason. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> (cherry picked from commit 93229f9)
16cfee9 to
8cac21f
Compare
buluoray
left a comment
There was a problem hiding this comment.
Approve at 8cac21f1f2093680f550b7c5557c325413cb9e7f.
The change is two expressions, one per lane: .github/workflows/fork-design-review.yml:768 and .github/workflows/fork-first-principles-review.yml:576. Each grep | head -n1 | sed | tr verdict capture becomes the single-pass awk … <<< "$summary". I checked it against source at this head.
Security (fork / workflow_run)
-
No trigger,
permissions:, job,if:,env:orshell:changes. Both lanes are stillworkflow_run-only, run from the default branch, with the same top-level permissions. Nothing here touchespull_request_target. -
No new
${{ … }}interpolation inrun:. The model output is still read from$summarythrough a here-string, and the awk program is a fixed single-quoted literal. -
No output injection: awk prints one record with every
\rremoved, soverdictcannot carry a newline into$GITHUB_OUTPUT. That is the same as before. -
Not fail-open. For every input that produced a value before, the new expression returns the same value: first anchored header, leading separator stripped, upper-cased. The tests check this on
plain,lowercaseandcrlf. The only inputs that behave differently are the two that used to abort the step:- No header: this now lands on
verdict="UNKNOWN", which the finalize step maps to neutral "review incomplete". An empty output maps there too. - Long review: the first header is now read from a complete read. The FP marker check
[FIRST-PRINCIPLES-REVIEWED] $HEAD_SHAstill runs after the capture, and so does the Design[DESIGN-REVIEWED]gate.
The long-review fix also closes a small fail-open in the old code. A long review carrying
BLOCKused to abort the step, so the post step (if: always()) published the generic notice instead of the…-Verdict: BLOCKline that readiness reads. That BLOCK was dropped. It now posts. - No header: this now lands on
-
mawk: the
[ \t\v\f\r]class insub()is the same one already running onubuntu-latestinfork-security-scope-review.yml:660andsecurity-scope-review.yml:1709.
Mutation checks (in a local worktree, then restored)
- I put back the old
grep | headcapture in both lanes. 6 of 14TestForkVerdictCaptureNeverAbortsAboveItsOwnFallback::test_every_ordinary_review_reaches_the_fallback_with_the_right_valuecases failed:no-header,header-shaped-proseandmany-headers, ×2 lanes. - I dropped the
^anchor in the Design awk. Two tests failed:[header-shaped-prose-fork-design-review.yml-…]andTestReviewLanesPublishOnlyTheReview::test_every_lane_trims_to_its_own_header. - Pristine head: 58/58 pass across the new class and the three re-pinned classes.
Bot lanes: FP PASS, Design PASS, GPT and Opus no findings; 86 check-runs green, 8 skipped.
- FP's deferred sibling is real but outside this PR.
fork-ux-review.yml:1279guards only the grep arm (|| true), and itshead -n1can still SIGPIPE underpipefail. It needs a follow-up. - Design's "verify the first fork PR after merge" is correct:
workflow_runruns main's yaml, so these tests are the only check before merge.
Non-blocking. Please fix before merge, because these land in main's history:
-
The squash commit body begins mid-sentence ("the two Security Scope lanes. Its two counted siblings…"). The original first line,
#12351 replaced a grep | head -n1 verdict capture with one awk pass in, began with#, and the cherry-pick dropped it as a comment. Please restore it in the squash message. -
The PR body and commit message overstate the symptom. "posts nothing at all — no comment" and "the comment that would have named the cause is never posted" are not accurate. The post steps are
if: always()(fork-first-principles-review.yml:630,fork-design-review.yml:811), so an aborted capture still posted the generic "no parseable verdict" notice, and the finalize step still published neutral "review incomplete". The real effects were:- a red job for a headerless review, with the same comment and check result as now;
- a lost verdict, BLOCK included, for a review over the pipe buffer.
Neither affects the fix itself.
Replaces #12517 by @leonlaiyc (same change, moved in-repo because the fork workflow-change guard blocks all bots on a fork PR under
.github/**, and the fork lane does not consume an override). Original authorship preserved on the commit; co-authored by the original author.Problem / Motivation
Goal: On a fork PR, make the Design Review and First Principles lanes always post their verdict (or a readable UNKNOWN), instead of silently posting nothing when the review has no header or is very long.
On a fork PR, the Design Review and First Principles lanes can finish their
model call, hold a perfectly good verdict, and post nothing at all — no comment,
no verdict output, and nothing in the job log naming why.
Both lanes capture their verdict with
inside a
run:block that opensset -uo pipefailand, having noshell:key,is executed by the runner as
bash -e. Two ordinary model outcomes make thatassignment's status non-zero, and
-ethen kills the step one line above thefallback written for exactly those outcomes:
grepmatches nothing and exits 1;pipefailpromotes that tothe pipeline's status.
head -n1exits after the first line and closes the pipe,so its producer dies of SIGPIPE, status 141.
Why it matters
The
verdict="UNKNOWN"default and the[ -n "$v" ]test directly beneath thecapture are a complete, fail-closed answer for a review that names no verdict.
Neither line runs. The outcome is not "UNKNOWN" but nothing: the lane writes no
verdict output, its status step reads an empty verdict, and the comment that
would have told a contributor what happened is never posted. A red either way,
but only one of them says why — and on a fork PR the contributor cannot re-run
the lane themselves.
The 141 arm is the worse one: a verdict is already in hand when the read ends
early, so an expression that merely suppressed the status would hand the lane a
verdict read out of a document the step never finished reading.
Not a goal
^…-Verdict:with no leading-whitespaceclass, so exactly the same set of headers counts as before. Accepting an
indented header is a separate question with its own contract argument (fix(ci): scope review must not abort on a headerless verdict #12351
settled it for the scope lanes); answering it here would mix a behaviour change
into an abort fix.
design-review.ymlandfirst-principles-review.ymlare nottouched. Their post-summary steps carry no
set -line, so an unmatchedgrep's status is masked by the pipeline's last command and the step cannot die
this way. Converting them would be a change with no defect behind it. (The
fork-ux-review.yml/ux-review.ymlcaptures already guard the grep arm with{ … || true; }; out of scope here, kept to what fix(ci): stop the fork advisory verdict captures aborting their own step #12517 touched.)added, renamed or removed. Only two expressions change, one inside one
existing step each.
What changed (motivation → approach → change)
Symptom → a fork advisory lane that reviewed fine posts no verdict and no
explanation.
Root cause → the capture is a pipeline in a step that turns a pipeline's
inner failure into the step's failure. Each of these steps already uses a
here-string twice — trimming the summary to the header, and checking for the
[…-REVIEWED]marker — each time under a comment saying a long summary must notmanufacture a SIGPIPE status under
pipefail. The capture sitting between themis the one place that rule was not applied.
Change → both captures become the one-
awk-pass form #12351 introduced:awkexits 0 whether or not the header is there, so the[ -n ]test belowdecides — as the code was always written to. There is no
exiton first match:the whole summary is read and the FIRST header is printed from
END, so averdict is never taken from a partial read. The character class spells out the
five characters
[[:space:]]matches inside a record, because the runners' awkis mawk.
Two existing pins moved as part of the diff:
TestFirstPrinciplesReviewandTestTheOutputDietTightenedAndKeptItsMachineReadLinesasserted the literalgrep -iE '^First-Principles-Verdict:', and_capture_stepused the sameliteral as a slice boundary. All three now key on an anchored reference to the
header rather than on the program that reads it — the property those pins were
about — because a pin keyed to one tool stops finding the capture the day the
capture changes tool and then measures nothing.
Backwards compatibility
Compatible. The capture returns the same value for every input that produced a
value before — the three verdict-carrying cases (
plain,lowercase,crlf)pass on pristine
origin/mainas well as on this branch, so they measure the oldexpression and the new one to be equal. The only inputs whose behaviour changes
are the two that produced no value and killed the step; they now produce no value
and let the step continue to its own fallback. No workflow trigger, permission,
job, step name, check-run name or output changes.
Tests
test/test_ai_review_workflows.py—TestForkVerdictCaptureNeverAbortsAboveItsOwnFallback, 14 tests over the twolanes. Each test extracts the step's own expression from the yaml and runs it
under
bash -ewith the step's ownset -uo pipefail, then asserts both theexit status and the captured value.
Six review kinds per lane:
no-header""header-shaped-prose""many-headers(60 000)PASSplainCONCERNSlowercaseBLOCKcrlfBLOCKPlus
test_the_step_still_carries_the_fallback_this_pin_is_about, which fails ifverdict="UNKNOWN"or the[ -n "$v" ]test is ever removed.Local run (this box, in-repo branch off fresh
origin/main): whole filetest/test_ai_review_workflows.py— 1123 passed; the new class alone — 14passed. Both edited workflow files re-parsed with
yaml.safe_load.Manual verification
The mechanism was reproduced in #12517 directly, running each lane's own capture
line under
bash -ewithset -uo pipefail: no-header → rc 1, 60 000 headers →rc 141, both one line above the
[ -n "$v" ]fallback.workflow_runlanesexecute the default branch's yaml, so this change is unexercised until after
merge; the first fork PR opened afterwards is the real end-to-end check.
Related Issues
None. This is the counted, unconverted half of #12351's First Principles review,
which named both
fork-design-review.ymlandfork-first-principles-review.ymlas the siblings that abort one line above their own
[ -n "$v" ]fallbacks.Pattern harvest
Rule candidate: review-prompt
Pattern: a step's preamble (
set -euo pipefail) makes a command's exit statusload-bearing, so an expression written to return "nothing found" instead kills
the step above the branch that handles "nothing found".
The prompt this proposes: when a fix converts a pipeline for exit-status reasons,
grep the sibling lanes for the same expression before calling the defect closed —
and when reviewing any
run:block that opens withpipefail, ask of every$( … | … )whether its "found nothing" status is a value the next line reads ora signal that kills the step first.
Checklist
feat|fix|docs|refactor|perf|test|chore|ci|build|revert: ...)