Skip to content

fix(ci): stop the fork advisory verdict captures aborting their own step - #12517

Closed
leonlaiyc wants to merge 1 commit into
kirodotdev:mainfrom
leonlaiyc:fix/fork-verdict-capture-abort
Closed

leonlaiyc wants to merge 1 commit into
kirodotdev:mainfrom
leonlaiyc:fix/fork-verdict-capture-abort

Conversation

@leonlaiyc

@leonlaiyc leonlaiyc commented Sep 21, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

On a fork PR, the Design Review and First Principles lanes can finish their
model call, hold a perfectly good verdict, and post nothing at all — no comment,
no verdict output, and nothing in the job log naming why.

Both lanes capture their verdict with

v="$(printf '%s\n' "$summary" | grep -iE '^Design-Verdict:' | head -n1 | sed -E '…' | tr … )"
[ -n "$v" ] && verdict="$v"

inside a run: block that opens set -uo pipefail and, having no shell: key,
is executed by the runner as bash -e. Two ordinary model outcomes make that
assignment's status non-zero, and -e then kills the step one line above the
fallback written for exactly those outcomes
:

  • no header — grep matches nothing and exits 1; pipefail promotes that to
    the pipeline's status.
  • a long review — head -n1 exits after the first line and closes the pipe,
    so its producer dies of SIGPIPE, status 141.

Why it matters

The verdict="UNKNOWN" default and the [ -n "$v" ] test directly beneath the
capture are a complete, fail-closed answer for a review that names no verdict.
Neither line runs. The outcome is not "UNKNOWN" but nothing: the lane writes no
verdict output, its status step reads an empty verdict, and the comment that
would have told a contributor what happened is never posted. A red either way,
but only one of them says why — and on a fork PR the contributor cannot re-run
the lane themselves.

The 141 arm is the worse one: a verdict is already in hand when the read ends
early, so an expression that merely suppressed the status would hand the lane a
verdict read out of a document the step never finished reading.

This is not a hypothetical shape. #12351 fixed precisely it in the two Security
Scope lanes; its First Principles review counted these two as the unconverted
siblings and named both paths. They are still on main at 1fa7f5ba5.

What changed (motivation → approach → change)

Symptom → a fork advisory lane that reviewed fine posts no verdict and no
explanation.

Root cause → the capture is a pipeline in a step that turns a pipeline's
inner failure into the step's failure. Each of these steps already uses a
here-string twice — trimming the summary to the header, and checking for the
[…-REVIEWED] marker — and each time under a comment saying, in the file's own
words, that a long summary must not manufacture a SIGPIPE status under
pipefail. The capture sitting between them is the one place that rule was not
applied. That is what makes this an oversight rather than a different judgement.

Change → both captures become the one-awk-pass form #12351 introduced:

v="$(awk '!found && tolower($0) ~ /^design-verdict:/ { sub(/^[^:]*:[ \t\v\f\r]*/, ""); gsub(/\r/, ""); value = toupper($0); found = 1 } END { if (found) print value }' <<< "$summary")"

awk exits 0 whether or not the header is there, so the [ -n ] test below
decides — as the code was always written to. There is no exit on first match:
the whole summary is read and the FIRST header is printed from END, so a
verdict is never taken from a partial read. The character class spells out the
five characters [[:space:]] matches inside a record, because the runners' awk
is mawk.

Three things deliberately do not change:

  • The anchor. It stays ^…-Verdict: with no leading-whitespace class, so
    exactly the same set of headers counts as before. Widening it to accept an
    indented header is a separate question with its own contract argument (fix(ci): scope review must not abort on a headerless verdict #12351
    settled it for the scope lanes, whose contract displays the header indented);
    answering it here would mix a behaviour change into an abort fix.
  • The same-repo design-review.yml and first-principles-review.yml. Their
    post-summary steps carry no set - line, so an unmatched grep's status is
    masked by the pipeline's last command and the step cannot die this way. I read
    both steps to confirm it rather than assuming symmetry. Converting them would
    be a change with no defect behind it.
  • The three here-strings already in these steps. Untouched.

Two existing pins had to move, and that is part of the diff rather than a
side effect.
TestFirstPrinciplesReview and
TestTheOutputDietTightenedAndKeptItsMachineReadLines asserted the literal
grep -iE '^First-Principles-Verdict:', and _capture_step used the same
literal as a slice boundary. All three now key on an anchored reference to the
header
rather than on the program that reads it — which is the property those
pins were about, and the reason the repo's own newer selector comments give: "a
pin keyed to one tool silently stops finding the capture the day the capture
changes tool, and then measures nothing."
The slice boundary takes both
spellings and cannot match the trim a few lines above it: the trim greps with
-q and its awk opens f ||, where the capture's opens !found &&.

Backwards compatibility

Compatible. The capture returns the same value for every input that produced a
value before — pinned, not asserted: the three verdict-carrying cases below
(plain, lowercase, crlf) pass on pristine origin/main as well as on
this branch, so they measure the old expression and the new one to be equal. The
inputs whose behaviour changes are the two that produced no value and killed the
step; they now produce no value and let the step continue to its own fallback.

No workflow trigger, permission, job, step name, check-run name or output changes.

Two open PRs touch the same three files and are worth naming so a reviewer is
not surprised by a textual conflict: #12201 and #12469 both edit the network
allowlist near the top of these workflows (lines ~68-160) and both append to the
end of test_ai_review_workflows.py. Neither touches a verdict capture -- I
grepped both diffs for Design-Verdict, First-Principles-Verdict, pipefail
and head -n1 and got zero hits -- so this is a rebase, not a competing fix.

Tests

test/test_ai_review_workflows.py —
TestForkVerdictCaptureNeverAbortsAboveItsOwnFallback, 14 tests over the two
lanes.

Each test extracts the step's own expression from the yaml and runs it under
bash -e with the step's own set -uo pipefail, then asserts both the exit
status and the captured value. Asserting the value alone would pass an expression
that returns the right answer and kills the step anyway; asserting the status
alone would pass one that never aborts and has stopped reading verdicts.

The expression is located by shape, and specifically by what the fallback
reads: _capture_line parses the variable name out of
[ -n "$v" ] && verdict="$v" and finds that assignment. Both steps also trim
their summary with a second assignment mentioning the same header, and a
first-match-wins selector silently measures that one instead — which it did in my
first draft, and the megabyte-long failure message is how I noticed.

Six review kinds per lane:

kind expected what it pins
no-header "" grep's exit 1 no longer takes the step down
header-shaped-prose "" the anchor still tells a header from a mention of one
many-headers (60 000) PASS SIGPIPE 141 gone; first header still wins
plain CONCERNS ordinary read unchanged
lowercase BLOCK the value is still upper-cased
crlf BLOCK no carriage return left inside the value

Plus test_the_step_still_carries_the_fallback_this_pin_is_about, which fails if
verdict="UNKNOWN" or the [ -n "$v" ] test is ever removed — without them the
whole class is measuring something else.

Red-before / green-after, whole file, -p no:randomly -n 4:

pristine workflows @ 1fa7f5ba5 this branch
test_ai_review_workflows.py 6 failed, 541 passed, 249 skipped 547 passed, 249 skipped

The red-before was run with the final test file against the two pristine
workflow files, so the 541 passes prove the moved pins are not what turns the
suite green — only the six abort arms are.

Gates on this head: flake8 clean, black --check clean,
scripts/check_black_formatting.py passed ("nothing in scope is unformatted
outside the baseline"), scripts/check_brand_name.py exit 0, and both edited
workflows re-parsed with yaml.safe_load.

Manual verification

The mechanism was reproduced directly on this box before any change, running each
lane's own capture line under bash -e with set -uo pipefail:

lane body rc fallback reached
fork-design-review no header 1 no
fork-design-review 2 000 headers 0 yes
fork-first-principles-review no header 1 no
fork-first-principles-review 60 000 headers 141 no

Stated exactly as measured: at 2 000 headers the SIGPIPE arm did not fire on
Git-for-Windows Bash, because a 64 KiB pipe buffer swallows that body whole. It
fires at 60 000, which is why the test fixture uses that size rather than the
2 000 #12351's own suite uses on Linux runners.

What is not verified here: workflow_run lanes execute the default branch's
yaml, so this change is unexercised until after merge. The first fork PR opened
afterwards is the real end-to-end check.

One process note, since this is a fork PR under .github/**: the Fork
Workflow-Change Guard
will flag it by design. Its own header says it is an
advisory check-run and not a merge gate, and that a maintainer who has reviewed
the change clears it with the allow-fork-workflow-change label. The diff adds
no job, step, trigger, permission or network allowance — it replaces two
expressions inside one existing step each, and updates the tests that read them.

Related Issues

None. This is the counted, unconverted half of the First Principles CONCERNS on
#12351 (merged), which names both paths: "fork-design-review.yml:666 and
fork-first-principles-review.yml:592 … so a headerless or many-header review
aborts them one line above their own [ -n "$v" ] fallbacks, exactly the defect
this PR names."

Pattern harvest

Rule candidate: review-prompt

Pattern: a step's preamble (set -euo pipefail) makes a command's exit status
load-bearing, so an expression written to return "nothing found" instead kills the
step above the branch that handles "nothing found".

Both lanes had the handling. Both lanes had already met this exact hazard twice in
the same step and written a here-string with a comment explaining why. The
surviving pipeline is what a reader skips because the fallback beneath it looks
like the answer.

The prompt this rule candidate proposes: when a fix converts a pipeline for
exit-status reasons, grep the sibling lanes for the same expression before
calling the defect closed — and when reviewing any run: block that opens with
pipefail, ask of every $( … | … ) whether its "found nothing" status is a
value the next line reads or a signal that kills the step first.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

🤖 Generated with Claude Code

kirodotdev#12351 replaced a `grep | head -n1` verdict capture with one awk pass in
the two Security Scope lanes. Its two counted siblings were left: the fork
Design Review and fork First Principles lanes read their header the old
way, inside a step that carries `set -uo pipefail`.

Both statuses that shape produces abort the step under the runner's own
`bash -e`, one line ABOVE the `[ -n "$v" ] && verdict="$v"` fallback
written for exactly those cases. A review with no header makes grep exit 1
and pipefail promotes it; a long review makes `head -n1` close the pipe and
its producer die of SIGPIPE with 141. The lane then writes no verdict
output at all, and the comment that would have named the cause is never
posted. Reproduced locally on both lanes: rc=1 with a headerless body,
rc=141 at 60k headers, and the fallback unreached in all four runs.

Each of these steps already uses a here-string twice, each time under a
comment saying a long summary must not manufacture a SIGPIPE status under
pipefail. The capture between them was the one place the rule was not
applied, which is what makes this an oversight rather than a judgement.

Convert both to the same awk expression kirodotdev#12351 used, with the anchor left
exactly as it was, so which headers count does not change. The same-repo
design and first-principles lanes are deliberately untouched: their steps
carry no `set -` line, so an unmatched grep's status is masked by the
pipeline's last command and they cannot die this way.

Two existing pins named `grep` as the program that reads the header. They
now assert an ANCHORED reference to the header instead, because a pin keyed
to one tool stops finding the capture the day the capture changes tool and
then passes while measuring nothing. `_capture_step`'s slice boundary takes
both spellings for the same reason.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@leonlaiyc
leonlaiyc requested review from a team and chenmingwei23 September 21, 2026 10:31
@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: action required A blocking check or review needs attention merge conflict Branch has merge conflicts with its base — author must resolve before merge labels Sep 21, 2026
@NicholasRBowers NicholasRBowers added needs-pr-triage PR scanner: awaiting automated triage needs-author-decision PR blocked on author input and removed needs-pr-triage PR scanner: awaiting automated triage labels Sep 25, 2026
@NicholasRBowers

Copy link
Copy Markdown
Contributor

🤖 Kiro Crew [operator: NicholasRBowers#a942f9ca]: I reviewed the blockers on this PR. The change itself looks right, but everything left is yours — no commit on this branch can move it.

The blocker: not one check has ever run. Every workflow on 93229f94 sits at action_required — CI, Fast Gate, Build, Code Review, all five AI review lanes, Cross-Platform Portability, Dependency License Gate, Internal Content Scan Gate, PR Scope, Screenshot Evidence, macOS Tests — since the PR opened on 21 September. PR Readiness reports it as "4 workflow(s) awaiting maintainer approval; none has run yet". The only jobs that have run are the three pull_request_target ones that execute from the default branch (Fork PR Description, Fork PR Label, and the readiness publisher).

That is the repo's "require approval for all external contributors" setting, which fork-workflow-guard.yml names in its own enforcement note as the real protection against a fork changing CI — the guard itself is advisory and is not a required check. So the decision needed is a maintainer pressing Approve and run, and separately whether this workflow change earns the allow-fork-workflow-change label. A rebase, a rerun, or a new push cannot produce that approval; a new push actively re-arms it, because the guard's strip-stale-override job invalidates any prior approval on synchronize.

The premise still holds — this is not superseded. I checked current main: both lanes still capture their verdict with the vulnerable pipeline, at fork-design-review.yml:698 and fork-first-principles-review.yml:606:

v="$(printf '%s\n' "$summary" | grep -iE '^…-Verdict:' | head -n1 | sed -E '…' | tr … )"

while fork-security-scope-review.yml:660 already carries the single-awk-pass form, with a comment saying in as many words that a grep matching nothing exits 1 and a head -n1 closes the pipe on a long review. So #12351 converted the Security Scope lanes and left these two behind, exactly as this PR's description claims.

Decisions I need from you:

  • Approve the workflow runs (and rule on allow-fork-workflow-change), or say the change should be re-homed as a same-repo branch. Until one of those happens there is no CI signal to drive and nothing for me to verify against — this PR cannot go green on its own.
  • Confirm you want the fix here rather than folded into a maintainer-authored commit. A fork PR touching .github/** also draws code-owner review through the catch-all * rule in CODEOWNERS, so it needs your eyes either way.

What is mechanical once you have ruled, and what I can take back if you want it: the branch is 629 commits behind and has exactly one conflict, in test/test_ai_review_workflows.py. The two workflow files merge clean against current main. That is the whole remainder.

When you've addressed these, the pipeline will re-assess on its next cycle.

@iamwhatever

Copy link
Copy Markdown
Contributor

Thank you for this fix, @leonlaiyc — the SIGPIPE/grep-exit-1 analysis of the fork verdict captures was spot on, and the test coverage is excellent.

Because this PR touches .github/** from a fork, the workflow-change guard blocks the review bots and the fork lanes do not consume an override, so it could not be driven to green in place. I opened an in-repo replacement that carries your commit (authorship preserved, you credited as co-author) rebased onto current main with the conflict resolved: #15845. It is now fully green — all review lanes and gates pass. Review feedback there (a ratchet-scope tightening on the re-keyed header pin and a prose-fixture fix) has been applied on top.

Closing this one in favor of #15845. Thanks again!

@iamwhatever iamwhatever closed this Oct 1, 2026
@github-actions github-actions Bot removed the readiness: action required A blocking check or review needs attention label Oct 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor) merge conflict Branch has merge conflicts with its base — author must resolve before merge needs-author-decision PR blocked on author input

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants