Repository navigation
fix(test): take the AI-review harness off a SIGPIPE-prone pipeline - #14426
Conversation
test_the_trim_and_count_execute_as_written built its script with `printf '%s\n' "$summary" | head -n1` and ran it under `bash -euo pipefail`. head exits on its first line and closes the read end, the producer's next write takes SIGPIPE, and pipefail promotes that 141 to the script's exit status -- failing `assert result.returncode == 0` with a message that reads like a review-lane problem. Whether the producer is still writing when head exits is a scheduling race, so the failure is rare and CI-only; seen on PR #14369's Backend Tests (3.12, 2) shard. This file already bans the shape where it ships: test_guard_function_is_byte_identical_across_all_lanes asserts no lane's guarded_comment_upsert contains `head -n1` and requires `| awk 'NR == 1'` instead, and _MANY_HEADERS records the measurement behind it. The harness now uses that same form. awk reads to EOF rather than exiting early, so the producer is never signalled. No assertion, fixture or expectation changed.
Design Review (Fable 5) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing to check. [DESIGN-REVIEWED] 81779b8 |
GPT 5.6 Review — ✅ no blocking findingsGPT 5.6 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
Opus 5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
Problem / Motivation
Goal:
Backend Testsstops going red at random on PRs that change nothing near the AI review workflows.test/test_ai_review_workflows.py::TestReviewLanesPublishOnlyTheReview::test_the_trim_and_count_execute_as_writtenfails intermittently in CI on branches whose diff touches neither that file nor.github/. Observed on PR #14369, shardBackend Tests (3.12, 2), job 108582066780:141is128 + SIGPIPE, and the captured stdout stops after the harness's first pipeline — so the script died in that pipeline, before the word count on the line below it ever ran.Why it matters
It is a false red on somebody else's PR. The failing assertion names a First-Principles verdict, so it reads as a review-lane problem in a file the author never opened; the author has to pull the shard log, notice the
141, and work out that the CI-only failure is the harness and not their change. Everything downstream of the shard (Coverage Merge,Coverage Gate,PR Readiness) reds with it, so it also masks the PR's real state until somebody reruns the job.Not a goal
head -n1. The existing assertion attest_guard_function_is_byte_identical_across_all_lanescovers the lane code, which is the code that ships; a second guard pointed at the test file would be a test of a test.awk … | wc -w | tr -d ' 'has no early-exiting consumer, so it cannot raise this signal.What changed (motivation → approach → change)
Symptom:
rc=141with the harness's own first line already on stdout.Root cause: the harness built its script with
printf '%s\n' "$summary" | head -n1, and ran it underbash -euo pipefail.head -n1exits as soon as it has one line and closes the read end; the producer's next write then takesSIGPIPE, andpipefailpromotes that141to the whole script's exit status, trippingassert result.returncode == 0. Whether the producer is still writing whenheadexits is a scheduling race, which is why this is rare and CI-only: the shard runs eight xdist workers on a loaded runner.This file already knows the hazard and already fixes it one way.
test_guard_function_is_byte_identical_across_all_lanesasserts that no lane'sguarded_comment_upsertcontainshead -n1— with the comment "never via a| head -n1inside the pipeline (SIGPIPE under pipefail)" — and requires| awk 'NR == 1'instead._MANY_HEADERSrecords the measurement behind it: the pipe closes on its producer somewhere "between 200 and 500" lines on Linux.Change: the harness now uses the same
| awk 'NR == 1', with a comment pointing at the assertion that holds the lanes to it.awkreads to EOF instead of exiting early, so the producer is never signalled and the race has nowhere to land. No assertion in the test changed; the fixture, the trim under test and every expectation are untouched.Backwards compatibility
Compatible: the change is inside one test's local harness string. No lane shell, no production code path, no fixture and no public contract is touched.
Tests
No new test. The change removes a race from an existing test's harness rather than adding behavior, and the case's own assertions (first line is the header, the word count excludes the collapsed inventory) already lock in what the harness is there to check.
TestReviewLanesPublishOnlyTheReview— 4 tests — passes.A regression test would have to win a scheduling race on demand, which is exactly what it cannot do reliably; the class is instead pinned where it ships, by the existing
head -n1assertion on the lane code.Manual verification
Demonstrated the mechanism and the fix's immunity directly, on Linux,
bash -euo pipefail, 200 runs each:grep -E … big.md | head -n1(2000 matching lines)141× 200grep -E … big.md | awk 'NR == 1'(same input)0× 200cat big.md | head -n10× 200The first two lines are the defect and the fix: an early-exiting consumer signals a still-writing producer, and
awk 'NR == 1'does not. The third is why this particular instance is rare rather than constant — a single large write can complete before the consumer exits, andprintfis a bash builtin doing one write, so its window is narrow. I could not make theprintfform fail locally (400 runs at the real fixture size, 200 at 2000 lines, all0); the CI job log above is the evidence for that instance, and its141admits no other producer.Also ran the case as written after the change:
TestReviewLanesPublishOnlyTheReview, 4 passed.Related Issues
No tracking issue. Surfaced on PR #14369, where it was one of the reds standing between that PR and a clean round.
Pattern harvest
Rule candidate:
semgrepPattern: a pipeline whose consumer exits early (
head -n<N>,grep -q,head -c<N>) underset -o pipefail, which turns the producer's SIGPIPE into the script's exit status. This repository already enforces the rule by hand for lane code, with one assertion per shape and a measured threshold; a semgrep rule over shell inside.github/workflows/**and over shell strings built intest/**would catch the next one without a new bespoke assertion.Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)