Skip to content

fix(test): take the AI-review harness off a SIGPIPE-prone pipeline - #14426

Merged
bolichen97 merged 1 commit into
mainfrom
fix/ai-review-harness-sigpipe
Sep 27, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
fix/ai-review-harness-sigpipe

Conversation

@bolichen97

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Backend Tests stops going red at random on PRs that change nothing near the AI review workflows.

test/test_ai_review_workflows.py::TestReviewLanesPublishOnlyTheReview::test_the_trim_and_count_execute_as_written fails intermittently in CI on branches whose diff touches neither that file nor .github/. Observed on PR #14369, shard Backend Tests (3.12, 2), job 108582066780:

E   AssertionError: First-Principles-Verdict: CONCERNS
E    +  where 141 = CompletedProcess(args=['/usr/bin/bash', '-euo', 'pipefail', '-c', ...],
                                     returncode=141, stdout='First-Principles-Verdict: CONCERNS\n', stderr='').returncode

141 is 128 + SIGPIPE, and the captured stdout stops after the harness's first pipeline — so the script died in that pipeline, before the word count on the line below it ever ran.

Why it matters

It is a false red on somebody else's PR. The failing assertion names a First-Principles verdict, so it reads as a review-lane problem in a file the author never opened; the author has to pull the shard log, notice the 141, and work out that the CI-only failure is the harness and not their change. Everything downstream of the shard (Coverage Merge, Coverage Gate, PR Readiness) reds with it, so it also masks the PR's real state until somebody reruns the job.

Not a goal

  • Does not change any review lane's own shell. The lanes were already held to the safe form; only this test's harness was not.
  • Does not add a guard asserting that the harness avoids head -n1. The existing assertion at test_guard_function_is_byte_identical_across_all_lanes covers the lane code, which is the code that ships; a second guard pointed at the test file would be a test of a test.
  • Does not touch the other pipelines in this harness. awk … | wc -w | tr -d ' ' has no early-exiting consumer, so it cannot raise this signal.

What changed (motivation → approach → change)

Symptom: rc=141 with the harness's own first line already on stdout.

Root cause: the harness built its script with printf '%s\n' "$summary" | head -n1, and ran it under bash -euo pipefail. head -n1 exits as soon as it has one line and closes the read end; the producer's next write then takes SIGPIPE, and pipefail promotes that 141 to the whole script's exit status, tripping assert result.returncode == 0. Whether the producer is still writing when head exits is a scheduling race, which is why this is rare and CI-only: the shard runs eight xdist workers on a loaded runner.

This file already knows the hazard and already fixes it one way. test_guard_function_is_byte_identical_across_all_lanes asserts that no lane's guarded_comment_upsert contains head -n1 — with the comment "never via a | head -n1 inside the pipeline (SIGPIPE under pipefail)" — and requires | awk 'NR == 1' instead. _MANY_HEADERS records the measurement behind it: the pipe closes on its producer somewhere "between 200 and 500" lines on Linux.

Change: the harness now uses the same | awk 'NR == 1', with a comment pointing at the assertion that holds the lanes to it. awk reads to EOF instead of exiting early, so the producer is never signalled and the race has nowhere to land. No assertion in the test changed; the fixture, the trim under test and every expectation are untouched.

Backwards compatibility

Compatible: the change is inside one test's local harness string. No lane shell, no production code path, no fixture and no public contract is touched.

Tests

No new test. The change removes a race from an existing test's harness rather than adding behavior, and the case's own assertions (first line is the header, the word count excludes the collapsed inventory) already lock in what the harness is there to check. TestReviewLanesPublishOnlyTheReview — 4 tests — passes.

A regression test would have to win a scheduling race on demand, which is exactly what it cannot do reliably; the class is instead pinned where it ships, by the existing head -n1 assertion on the lane code.

Manual verification

Demonstrated the mechanism and the fix's immunity directly, on Linux, bash -euo pipefail, 200 runs each:

pipeline exit statuses over 200 runs
grep -E … big.md | head -n1 (2000 matching lines) 141 × 200
grep -E … big.md | awk 'NR == 1' (same input) 0 × 200
cat big.md | head -n1 0 × 200

The first two lines are the defect and the fix: an early-exiting consumer signals a still-writing producer, and awk 'NR == 1' does not. The third is why this particular instance is rare rather than constant — a single large write can complete before the consumer exits, and printf is a bash builtin doing one write, so its window is narrow. I could not make the printf form fail locally (400 runs at the real fixture size, 200 at 2000 lines, all 0); the CI job log above is the evidence for that instance, and its 141 admits no other producer.

Also ran the case as written after the change: TestReviewLanesPublishOnlyTheReview, 4 passed.

Related Issues

No tracking issue. Surfaced on PR #14369, where it was one of the reds standing between that PR and a clean round.

Pattern harvest

Rule candidate: semgrep
Pattern: a pipeline whose consumer exits early (head -n<N>, grep -q, head -c<N>) under set -o pipefail, which turns the producer's SIGPIPE into the script's exit status. This repository already enforces the rule by hand for lane code, with one assertion per shape and a measured threshold; a semgrep rule over shell inside .github/workflows/** and over shell strings built in test/** would catch the next one without a new bespoke assertion.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable) — N/A, no documented behavior changes
  • No secrets, credentials, or internal references in the diff

test_the_trim_and_count_execute_as_written built its script with
`printf '%s\n' "$summary" | head -n1` and ran it under `bash -euo
pipefail`. head exits on its first line and closes the read end, the
producer's next write takes SIGPIPE, and pipefail promotes that 141 to
the script's exit status -- failing `assert result.returncode == 0` with
a message that reads like a review-lane problem. Whether the producer is
still writing when head exits is a scheduling race, so the failure is
rare and CI-only; seen on PR #14369's Backend Tests (3.12, 2) shard.

This file already bans the shape where it ships:
test_guard_function_is_byte_identical_across_all_lanes asserts no lane's
guarded_comment_upsert contains `head -n1` and requires `| awk 'NR == 1'`
instead, and _MANY_HEADERS records the measurement behind it. The
harness now uses that same form. awk reads to EOF rather than exiting
early, so the producer is never signalled.

No assertion, fixture or expectation changed.
@bolichen97
bolichen97 requested a review from a team as a code owner September 27, 2026 08:47
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Sep 27, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — ✅ PASS

Design-level review of 81779b8620da449778aff889f832b69811d6d178 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Nothing to check.

[DESIGN-REVIEWED] 81779b8

@github-actions

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ✅ no blocking findings

GPT 5.6 completed its review of 81779b8620da449778aff889f832b69811d6d178 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 81779b8

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 81779b8620da449778aff889f832b69811d6d178: <one-sentence reason>

@github-actions

Copy link
Copy Markdown
Contributor

Opus 5 Review — ✅ no blocking findings

Reviewed 81779b8620da449778aff889f832b69811d6d178 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 81779b8

Verdict parsed from the review's SHA-scoped output markers for commit 81779b8620da449778aff889f832b69811d6d178.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 81779b8620da449778aff889f832b69811d6d178: <one-sentence reason>

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Sep 27, 2026
@bolichen97
bolichen97 merged commit 8abbf3a into main Sep 27, 2026
96 checks passed
@bolichen97
bolichen97 deleted the fix/ai-review-harness-sigpipe branch September 27, 2026 09:24
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Sep 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant