Skip to content

fix(ci): retry a review lane's marker-comment write and stop claiming a lost one - #12957

Merged
bolichen97 merged 1 commit into
mainfrom
fix/reviewer-marker-publication-retry-12952
Sep 24, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
fix/reviewer-marker-publication-retry-12952

Conversation

@chenmingwei23

@chenmingwei23 chenmingwei23 commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

What is the problem?

A review lane can earn a verdict for the PR's current head and still fail to
publish it, while reporting success.

Each lane's verdict lives in one marker comment, [<LANE>-REVIEWED] <sha>.
guarded_comment_upsert() gates the write on two reads, the bot's existing
comment id and the PR's current head sha, and withholds if either cannot be
confirmed. Withholding on an unreadable head is correct: the comments API has no
If-Match, so the write is last-writer-wins and a wrong one buries a live
verdict. Three things after that point are not correct.

The two gating reads retried 3 times with 1s/2s backoff, about 3 seconds of
tolerance, against App-wide API exhaustion that lasts minutes. When they failed,
the function returned 0, the job concluded success, and nothing retried.

The write itself was attempted exactly once, under || true, with its
Updated existing <lane> comment #<id> line printed unconditionally on the next
line. A lost PATCH therefore announced itself as a published verdict.

A withheld verdict was a plain echo in a job log, so a lane that published and
a lane that withheld were indistinguishable from outside.

Why this issue matters to the user

The marker comment is the only thing a freshness verifier reads. A verdict that
does not reach it leaves the PR indistinguishable from one no lane ever
reviewed: prepare-pr's pr_status.py exits 20 on a PR whose check-runs, PR Readiness status and mergeable state all pass, and that refusal is correct,
because the markers really are stale.

The only recovery was manufacturing a new head. That re-rolls every
non-deterministic review lane and discards the verdicts already earned, so the
cheapest way out of a lost write cost more than the review it lost. A reviewer
lane that fails to publish is worse than one that fails red: the gate stays
green, the marker stays old, and every prepare-pr and babysit loop reads
BLOCKED with nothing to act on.

How our fix solves it

The symptom is a stale marker. The marker is stale because the write never
landed. The write never landed because it got one attempt and no verification,
and the reads gating it gave up in about 3 seconds. So the fix is at the
publication chain, and nothing in pr_status.py changes: a stale marker is
still read as stale.

retry_comment_write() is one publication primitive. Six attempts, 5s linear
backoff, about 75s, and it returns the outcome instead of swallowing it. The
caller owns the wording, because only the caller knows which body it was
publishing. On success it prints what it printed before; on an exhausted budget
it emits an ::error:: annotation naming the head whose verdict is unpublished.

Both gating reads move to the same budget, since one exhaustion window fails
reads and writes alike, and the two withhold notices become ::warning::
annotations so a withheld verdict is distinguishable from a published one.

The two write kinds are not equally safe to repeat

A PATCH is idempotent: same URL, same body, so repeating it changes nothing.
A create is a POST and is not. GitHub can accept it and still lose the ack, so
a blind second attempt plants a second marker comment. Every id lookup in
these lanes selects exactly one comment, so a later human override patches only
that one while the duplicate keeps its own [BLOCK-MERGE] line, and
pr_status.py stays blocked until somebody finds and deletes the extra comment
by hand. That is the same gate-stranding this change exists to remove, reached
from the other side.

So a create may only repeat once it can RECOGNISE its own work, and the needle
is the current head's stamp. Only a run for this head writes <stamp> <sha>, so
a comment left by an older head cannot satisfy it. Marker presence cannot be the
needle: every body these lanes write starts with the lane marker, so a stale
comment would answer "landed" for a verdict that was never published, and the
retry that would have published it would be abandoned, which is this same defect
class arriving from a third direction.

A PATCH passes "" "" and repeats freely. A create passes the current head's
stamp and the body it is posting, and before each repeat the primitive re-queries
for that stamp. A body carrying no such stamp (a review-failure notice) has
nothing a later query can recognise, so it gets one attempt: for it a lost ack
and a rejection are indistinguishable, and repeating could duplicate it
undetectably.

An unreadable confirmation stops the create rather than repeating it, because
the two outcomes are not symmetric. A duplicate strands the gate with no
automatic recovery; an unpublished verdict is annotated, leaves the gate honestly
red, and a re-run fixes it. That is the same rule the head-confirmation read
already follows: the destructive half must not proceed on an unknown.

An unpublishable verdict is an infrastructure failure, not a BLOCK verdict, so
the advisory lanes still exit 0 and the annotation carries the signal. The one
exception is codex-review.yml's create, which has always been bare and failed
the step; the retry is added there without softening that, so only a genuinely
unpublishable verdict still exits non-zero.

Which lanes, and why that set

The set is every lane that publishes a verdict, which is wider than the
three lanes that went stale:

Path Lanes
Shared guarded_comment_upsert design-review, first-principles-review, security-scope-review, ux-review, and the six fork-* lanes
Divergent, same-repo claude-review.yml (Opus 5), codex-review.yml (GPT 5.6)

Design and First Principles went stale through the shared function's withhold
path. Opus went stale through claude-review.yml, which has no guard at all; it
is the lane whose log said it updated a comment that was never updated. GPT
carried the identical suppressed-write shape and escaped only because
codex-review.yml has an extra edited trigger that an unrelated body edit
re-fired. Fixing the shared function alone would have left the same defect in
the one lane whose recovery path everything else depends on.

retry_comment_write is byte-identical in all twelve, pinned by a test, the way
guarded_comment_upsert already is in ten.

Two questions, two needles

A create is only safe once the slot is known empty, and only safe to repeat once
the previous attempt is known not to have landed. Those are different questions
and they need different needles.

The slot is every comment starting with the lane marker, whatever head it
carries, because that is what each lane's own id lookup selects. A caller reaches
a create whenever its own lookup could not answer, so an occupant it could not see
is exactly when a POST puts a second comment in a slot that holds one. The
occupancy read therefore matches startswith on the lane marker, before the first
write:

The occupancy read says Action Reported as
Slot empty post published
Slot occupied post nothing that comment's id, and that a re-run updates it
Cannot be read post nothing neither claim, with the cost of each named

Landing is a different question, answered after the write by contains on the
current head's stamp, which only a run for this head writes. The lane marker
cannot answer it: an older head's comment carries the marker too, so matching on
it would report a verdict published that never was. Since the slot was confirmed
empty first, a stamp found afterwards is this run's own landed write.

A PATCH names a known id and is idempotent, so it passes an empty slot and an
empty stamp and takes neither check.

An occupied slot is refused, and a stranded verdict goes red

The slot holds Action Returns The lane
Nothing post, retrying a transient failure 0 green, Published
A verdict for THIS head post nothing 3 green, named as another run's
Any other comment post nothing 2 ::error::, and see the lane table below
A comment that arrived during the backoff, not this run's stop retrying 2 ::error::, same
An answer that cannot be read post nothing 1 ::error::, same
A newer head, re-confirmed before each repeat stop retrying 4 green, named as that head's to publish

One comment answers for this lane, so every alternative to refusing loses
something no run gets back. A second comment beside the occupant is the one a
later human override cannot reach. Overwriting the occupant can discard a verdict
this run has no standing to judge: a concurrent run's for the same head, or the
CURRENT head's when this run's own head is already superseded.

Refusing is only half an answer, though, because a lane that publishes nothing and
says nothing hides exactly the stranding this change exists to fix -- the PR reads
as unreviewed for its own head with every check green and nobody told. So every
such case emits an ::error:: annotation naming the head whose verdict is not on
the PR. The one occupant that costs the PR nothing is the slot already carrying a
verdict for this head: the stamp readers find the head they are looking for, so
status 3 keeps that case distinct and no caller reports another run's write as its
own.

Whether that also fails the lane depends on which lane it is

Lane A failure is attributed as An unpublished verdict
Design Review, UX Review, First Principles Review (BLOCK), a judged-wrong design annotation only, lane stays green
Opus 5 Review, GPT 5.6 Review plainly, clearable by a head-scoped override annotation, and the lane fails
pr-readiness.yml names exactly those first three lanes for the (BLOCK)
attribution, so failing one of them for a write that never landed manufactures a
design verdict nobody reached. Its own comment says of the rest that a plain
(failure) is the only attribution true of both a real verdict and a run that
measured nothing, "identical to the Design / Opus / GPT lanes, and cleared by a
one-comment /ai-review override".

The two same-repo reviewer lanes are fail-CLOSED, and their gate reads the captured
review text rather than the PR. So a verdict that missed the slot leaves the check
green while the PR presents an older head's verdict -- and codex-review.yml already
exits non-zero in one arm of that same if/elif chain, under a comment saying an
unpublishable verdict fails the step rather than passing quietly. Both lanes now
follow one rule: an outcome that leaves this head's verdict out of the slot exits
non-zero, which is four arms each. Status 4 stays green, because the run for the
current head publishes the verdict that counts.

Status 3 divides the two classes as sharply as a failed write does. An earlier run
for this SAME head already holding the slot costs an advisory lane nothing it can
act on. A fail-closed lane loses the fresh sample the re-run was asked for, and the
earlier verdict keeps whatever it said, a blocking line included -- so that verdict
goes on gating the PR while the lane reports green. The two fail-closed lanes
therefore name the occupant and fail; the ten shared lanes keep it green. Nothing
writes into the slot in either case: replacing what another run put there needs
standing the primitive does not have, which is why #13100 owns that question.

This happened in production on this PR while it was being reviewed, which is how the
distinction got measured rather than argued. On head 14350d53e7 the Opus lane
produced a clean verdict, then lost every one of the six attempts at its comment
lookup AND every one of the six at the slot read to an API window, published
nothing, and concluded success -- leaving the slot holding a verdict for a head two
pushes back. Nothing in CI caught it: the required readiness status reads check-run
conclusions and carries no stamp-freshness read at all, so the reader that sees a
head with no verdict is prepare-pr's own marker evaluation, which a human runs.
That is also why the shared lanes' annotation is not a lesser channel dressed up as
a choice: for an advisory lane it is the only honest one available.

The retry re-reads the whole slot rather than only bodies carrying this run's
needle. The slot was confirmed empty before the first write, so anything there
during the backoff arrived meanwhile: this run's own write whose ack was lost, or
a run for a different head publishing while this one waited. Reading only this
run's needle made the second invisible, and the retry then posted beside it.

A repeat re-confirms the head, and what that still cannot cover

The slot re-read protects a create. A replace has nothing equivalent: the caller
names the comment id and the primitive never reads that comment's body, so there
is no shape to compare and the repeat is blind. If the PR moves to a newer head
between two attempts, the run for that head owns the comment, and the second
attempt puts a superseded body back over a current verdict.

The head is the standing a replace rests on, so it is re-confirmed before every
repeat, on the same terms as the write it gates. A head that has moved stops the
loop and answers 4, which is not an error: the run for the current head publishes
the verdict that counts, so nothing is missing from the PR. An unreadable head
stops the loop too, because repeating is the destructive half and must not proceed
on an unknown -- what stopping costs is the recoverable side, an unpublished
verdict that is annotated and that a re-run publishes. The order inside the loop
matters and is pinned: the landing read runs first, so a write whose ack was lost
is still reported as landed rather than as a superseded no-write.

Residual, stated rather than implied. Two runs on the SAME head are alike to a
head check, so a same-head run that replaces the comment between two attempts has
its body overwritten by the repeat. Both bodies are verdicts for that head, so
every stamp reader finds the head it looks for and no gate is stranded; what is
lost is which sample is displayed. Telling the two apart needs a body pre-image or
a conditional write, and this API offers neither an If-Match nor an edit history,
so it cannot be closed inside this chain.

One same-head body is an exception, because losing it is not a display question: an
accepted human override REPLACES this slot's comment, and a verdict restored on top
of it puts back the block the override cleared, which no later run undoes. So a
repeat re-reads the comment it is about to write and stops when that comment says a
human overrode the lane. That is one bounded read of one id, not a compare-and-set:
it keeps no pre-image and recognises nothing but the override note, so the ordinary
same-head residual above stands unchanged. An unreadable answer stops the repeat on
the same terms as an unreadable head.

An accepted override carries its own marker

One body in this slot is authoritative for the head and carries no reviewer stamp:
the human override. Named by the stamp, it was classified a notice and left out,
so an accepted override left a standing block in the slot with nothing later to
recover it. Named by the bare head, it matched the blocking verdict it was posted
to clear, and the slot read then read that verdict as this run's own write and
reported the override as already published. So an override body emits, and is
matched by, a marker only an override carries: [OPUS-OVERRIDE] <sha> and
[GPT-OVERRIDE] <sha>, on the same head-scoped shape as the verdict stamps.

The lookup that gates the write is retried too

A caller may only replace the lane's comment if its own lookup succeeded -- that
read is what tells it the comment it found is the one to update. Two lanes looked
theirs up with a single attempt that swallowed its own error, so an ordinary flake
answered "no comment", routed a live verdict into the create path, and met the
refusal above. The result was a lane publishing nothing for a head the PR then
read as unreviewed: the same stranding by a different door, and worse than the
duplicate it replaced, because a duplicate at least left a verdict on the PR.
Those lookups now get the widened budget every other gating read in this chain
has, and they report their failure instead of hiding it.

One contradiction resolved, not a new trade

The withhold branch already refused to create when the lookup errored, on the
grounds that "a marker planted is not undone by a later run". Ten lines below, the
verdict branch created anyway, on the grounds that "a duplicate comment is
recoverable and an unposted verdict is not". Both cannot hold, and the second is
the false one: clearing a duplicate takes a person finding and deleting a comment,
while a withheld verdict is published by re-running the lane. The same unknown now
gets the same answer in both branches, and the prose asserting otherwise is
corrected in the same change.

Where the occupancy read answers cleanly and the slot is empty, a completed
verdict publishes exactly as before even though the caller's lookup failed, which
test_completed_verdict_creates_once_a_duplicate_is_ruled_out pins.

One thing needs a human, not a lane. test_completed_verdict_creates_when_the_lookup_fails
was a named pin, and this replaces it. The Design lane's advisory CONCERNS asks the
merger to ratify that flip rather than letting the change carry it silently, which
is right, so it is stated here: merging this accepts that a review lane withholds a
verdict it cannot publish safely, instead of publishing one that may strand the
merge gate. First Principles independently confirmed the unrecoverability premise
against pr_status.py, which blocks on ANY bot comment carrying [BLOCK-MERGE] for
the current head while every lane's id lookup selects exactly one comment. Security
Scope classified 23 candidates across three platforms and confirmed 0 newly
refused
, naming the same window as the one behavioural loss and gh run rerun as
its recovery.

The same defect, reported separately as #13011

#13011 reports Opus 5 Review concluding success while the PR carries no
[OPUS-REVIEWED] stamp for its head, so pr_status.py answers
stale reviewer stamp(s) on four of four sampled PRs. That lane's publication
path on main is a suppressed write followed by an unconditional claim:

gh api --method PATCH "repos/$REPO/issues/comments/$existing" \
  --field body="$(cat /tmp/claude-summary.md)" >/dev/null || true
echo "Updated existing Opus 5 summary comment #$existing"

So the PATCH's rejection is discarded, the log says the update happened, the
comment keeps an older head's stamp, and the lane stays green. That is this
change's own root cause, on the one lane that can prove the verdict existed:
Gate on Opus 5 review runs always() and fails closed unless
[OPUS-REVIEWED] $HEAD is in the output, so a success conclusion on an ordinary
PR means the review completed for that head and the write is where the stamp was
lost. (Two conditions pass that gate without a verdict, a recorded human override
and a dependabot PR, and neither is present on the four sampled.)

Both of that lane's writes now go through retry_comment_write, so a rejection is
retried on the widened budget and, if it persists, reported as
::error::Opus 5 ... could not publish it; the success line prints only for a
confirmed write. What this does not do is guarantee the write lands. A rejection
that is deterministic rather than transient -- an oversized body is the candidate
that would explain four of four -- survives every retry. It converts a silent
stale stamp into a named failure carrying GitHub's own reason, which is what
telling those two cases apart needs, so #13011 stays open for that measurement
rather than being closed here.

Deliberately out of scope

Some of these steps have other marker writes, for human-override notes and for
skip and no-contract notices, that still carry the one-attempt || true shape.
They are a separate defect with the same symptom: they sit before their step's
early exit 0, above the primitive's definition, so covering them means
hoisting the definition through twelve heterogeneous step preambles. That belongs
in its own change rather than folded in here, and the test that pins this
contract says so where it scopes itself to the verdict writes.

A re-fire entry that needs no new head, a workflow_dispatch or a sweep shaped
like pr-readiness-sweep.yml, is also not here. It is a recovery mechanism for
verdicts already lost, not a fix for losing them, and it is a new workflow rather
than a change to this chain.

Replacing an occupied slot is not here either, and #13100 tracks it. Deciding that
this run's body should displace a comment another run put there needs standing this
primitive does not have -- it knows the id, not what the occupant says or which run
wrote it -- so the decision belongs to a caller that can establish it. Refusing is
the conservative half of that trade and is loud: the verdict is annotated and a
re-run publishes it.

What tests we did

test/test_ai_review_workflows.py, 945 passed, run with an explicit -n0.

31 red against the head before this one: the override guard on an occupied target and on an unreadable one, the per-repeat target read inside the lost-write test, across ten lanes each, plus the primitive's identity pin. Four uniform mutations across all twelve lanes were each caught with none uncaught: the guard removed, an unreadable target treated as safe, the guard reading the target but never recognising an override, and the override outcome reported as this run's own publication.

21 red against the head before that one: the lane-class contract, plus the two behavioural tests that pin how an advisory lane reports an earlier same-head run's occupancy, across ten lanes each. Four uniform mutations were each caught with none uncaught: the discarded fresh sample going quiet in either fail-closed lane, a superseded run treated as a loss, and the same failure pushed into an advisory lane whose failure readiness reads as a judged BLOCK.

One red against the head before that one: the contract that pins which lane class exits on a verdict that missed the slot, which also asserts the shared function carries no exit in any arm. Four uniform mutations were each caught with none uncaught: the occupied-slot arm going quiet in a fail-closed lane, the replace arm going quiet there, a superseded run treated as a loss, and an exit 1 added to a shared lane.

62 red against the head before that one, over eight contracts: the retry's head re-confirmation on a moved head and on an unreadable one, its per-repeat read count, the three cases that must leave an advisory lane green rather than fail it, the override body's own marker, and the primitive's identity pin. Four uniform mutations across all twelve lanes were each caught with none uncaught: the head never re-confirmed before a repeat, an unreadable head treated as still current, the re-confirmation gated on the needle so only the create is covered, and the re-confirmation moved above the landing read so a landed write reports as a superseded no-write.

Refusing an occupied slot: 31 red against the head that wrote into one, being test_an_existing_current_head_comment_is_never_duplicated, test_a_superseded_head_comment_is_refused_too and test_a_concurrent_runs_comment_is_not_reported_as_this_runs_write across ten lanes each, plus the primitive's identity pin. Four uniform mutations -- the pre-write occupant answering success, the retry-time occupant answering success, the occupant check never firing, and an occupied slot overwritten again -- were each caught, none uncaught.

The needle's ordering and the retry's slot read: 2 red against the head that had neither, being the call-site contract's order assertion and the primitive's identity pin. Three uniform mutations -- the needle computed above the classification again, the retry read matching only this run's needle, and any occupant reported as this run's own landed write -- were each caught, none uncaught.

One answer for an occupied slot: 12 red against the head that had two, being test_a_stale_head_comment_is_replaced_not_refused across ten lanes, the primitive's identity pin, and the call-site contract that pins the override needle. Four uniform mutations -- an occupant never written, a failed update reporting success, the update turned back into a create, and the override body stripped of its head-scoped needle -- were each caught, none uncaught.

Writing into an occupied slot rather than stopping: 31 red against the head that stopped, being test_an_existing_current_head_comment_is_never_duplicated, test_a_concurrent_runs_comment_is_not_reported_as_this_runs_write and the new test_a_slot_update_that_cannot_land_is_not_reported_as_published across ten lanes each, plus the primitive's identity pin. Three uniform mutations across all twelve lanes -- the occupant preserved again, a failed update reporting success, and the update turned back into a create -- were each caught, none uncaught.

The occupant-is-two-answers change: 31 red against the head that carried the single answer, being test_a_stale_head_comment_also_occupies_the_slot, test_an_existing_current_head_comment_is_never_duplicated and test_a_concurrent_runs_comment_is_not_reported_as_this_runs_write across ten lanes each, plus the primitive's identity pin. Three uniform mutations across all twelve lanes -- the stale branch returning success, the slot read dropping the stamp field, and the no-stamp branch moved back above the slot question -- were each caught, none uncaught.

An earlier head carried four reds this change did not cause, all owned by main
and tracked as #13005: Backend Tests (3.12, 7) and Backend Tests (Windows) (7)
on test/test_crew_log_exc_info_sites.py, the Coverage Gate downstream of them,
and Frontend Lint & Type Check on a restyle ratchet this diff has no frontend
file to affect. Their fixes are #13007 and #13008, both merged, and this branch is
rebased onto a main that has each of them as an ancestor, so those lanes now
measure this diff.

Twelve reds on that same head were one API rate-limit window, not verdicts:
API rate limit exceeded for installation at 05:31:43Z, with every reviewer lane
failing closed on its own inability to read this PR's comments. The rebase re-rolls
them rather than a re-run spending into an unrecovered window.

Red-on-base first. The new and updated tests against unmodified base workflows:
54 failed, 58 passed. Every contract is red without the fix, including both
lost-write tests across 10 lanes each, the no-unconditional-success-claim test
across all 12, the primitive identity test, and the two widened read-budget tests
across 10 lanes each.

Each reviewer finding was proved the same way, against the head that carried it.
The non-idempotent-create finding: 12 failed, 111 passed, the failures being
test_a_create_whose_ack_was_lost_is_not_posted_again across 10 lanes plus two
contract tests. The stale-marker confirmation finding: 12 failed, 121 passed,
the failures being test_a_stale_marker_never_confirms_an_unpublished_verdict
across 10 lanes plus the same two contract tests. The stale-marker test drives
the exact interleaving the reviewer named: STUB_COMMENT_FAIL_FIRST fails only
the first N comment lookups, so the id lookup errors, the step takes the create
path, the POST fails, and the confirmation query then succeeds with an older
head's comment on the PR.

The occupancy-needle finding: 42 failed against the head that carried it,
being test_an_existing_current_head_comment_is_never_duplicated,
test_a_stale_head_comment_also_occupies_the_slot,
test_a_concurrent_runs_comment_is_not_reported_as_this_runs_write,
test_an_unreadable_duplicate_check_refuses_the_write and the updated lost-ack
test across 10 lanes each, plus the identity pin and the call-site contract. The
stale-head test is the crisp one: a comment for an OLDER head on the PR,
STUB_COMMENT_FAIL_FIRST=6 so the caller's lookup errors on all six attempts and
it takes the create path, and nothing failing the write, so an unguarded POST
succeeds and sits a second comment beside it. It asserts the create file does not
exist at all. Two tests hold the ordinary path working:
test_a_first_write_still_happens_when_the_slot_is_provably_empty and
test_completed_verdict_creates_once_a_duplicate_is_ruled_out.

The behavioural tests run each lane's real step bash with a stubbed gh,
injecting the failure. STUB_PATCH_FAIL and STUB_CREATE_FAIL fail every write
attempt and record the count, and record no accepted body, so a test cannot
assert on a body the API never took. STUB_CREATE_LANDS emulates the lost-ack
partial failure: GitHub accepts the POST and the CLI still reports failure, with
the body written into the finder fixture so the step's own confirmation query
sees it through real jq. The tests check the lane does not print
Updated existing / Created new, emits an ::error:: naming the head, posts
exactly once when an ack is lost, and still exits 0. A sleep shim on the stub
PATH records the backoff schedule (5 10 15 20 25) instead of waiting it out, so
the lanes keep their real production budget with no test-only knob in the
workflow.

Mutation proof, each mutation applied uniformly across the lanes so the
byte-identity pin stays satisfied and only behaviour breaks. Four on the retry
budget and the success claim: budget shrunk to one attempt, a write failure
announced as a published update, the primitive always reporting success, and every
lane reverted to the bare suppressed write. Four on the occupancy check: the
occupancy read keyed on the head stamp instead of the lane marker, an occupied slot
ignored, an unreadable answer treated as an empty slot, and the check skipped
entirely. All eight caught by the behavioural tests, zero uncaught.

Gates: scripts/check_comment_history.py self-test then the gate itself
(passed), prepare-pr preflight.py (READY), push_guard.py --base main --max-ahead 2 (SAFE TO PUSH), and all twelve workflows parse as YAML.

Pattern harvest

Rule candidate: semgrep

Pattern: a mutating API call whose failure is suppressed with || true,
immediately followed by an unconditional success log line. The suppression and
the claim are individually defensible and only wrong together, which is why
review kept reading past them. The shape is mechanically detectable in shell:
a command ending || true whose next statement is an echo asserting the write
happened. This PR removes five instances of it; the same shape remains in the
override-note and skip-notice writes named under "Deliberately out of scope",
which is evidence the class is worth a rule rather than twelve hand fixes.

Second rule candidate: review-prompt

Pattern: a retry loop wrapped around a non-idempotent request. Adding a retry is
usually read as strictly safer, so the review question "is this request safe to
repeat, and how would a repeat be detected" is not asked. It should be asked
whenever a retry is introduced around a POST, since the failure it creates
(a duplicate that strands a gate) can be worse than the one it fixes. Two rounds
of this PR's own review were spent on that question: first that the POST could
duplicate, then that the confirmation added to prevent it could false-positive on
a stale comment. A reviewer caught both, which is the argument for asking the
question in the prompt rather than per PR.

Any other suggestions on the work

claude-review.yml still has no head-confirmation guard, so it can publish a
verdict for a superseded head, which is the loss the shared guard exists to
prevent arriving from the other direction. Converging it onto
guarded_comment_upsert changes that lane's withhold semantics and deserves its
own review; this change only makes its write reliable and its log honest.

The retry budget is calibrated to one observed window. If a future exhaustion
outlasts 75s the annotation will now say so plainly, which is the input needed to
retune it.

Closes #12952

@chenmingwei23
chenmingwei23 requested review from a team and pepmach September 23, 2026 00:52
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Sep 23, 2026
@github-actions

github-actions Bot commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — 🟡 CONCERNS

Design-level review of 1e51676c739575b42dfc3c3c9fdfc0ec74e837b8 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: CONCERNS

Flips the named pin test_completed_verdict_creates_when_the_lookup_fails (#8350) from publish-on-unknown to withhold-on-unknown — a human must ratify that trade at merge.

Watch

  • The old pin's docstring calls publish-on-failed-lookup "the asymmetry that makes the guard safe in both directions"; this PR replaces it, arguing the recoverability claim was false (a duplicate needs manual deletion, a withheld verdict needs one re-run) and citing the base workflow's own contradictory withhold branch. The argument is sound and the flip is surfaced in the description rather than smuggled, but a pin that names the behaviour it forbids is a decision until a human says otherwise — and the new residual is real: a completed verdict can now go unpublished (annotation-only on advisory lanes), with recovery still a human noticing and re-running until the out-of-scope re-fire mechanism exists.
    Clears when: the merger explicitly acknowledges accepting withhold-over-publish (per the description's own request), or the re-fire/sweep follow-up lands making a withheld verdict self-recovering.

Suggestions

  • The byte-identity pin now covers two functions (~600 duplicated bash lines × 12 files), and the PR's own out-of-scope note says extending coverage "means hoisting the definition through twelve heterogeneous step preambles" — that cost is caused by the duplication; a follow-up extracting the primitives into one checked-in script (fork lanes run base-ref workflows, so a base checkout serves it) would retire the pin treadmill before the declared follow-ups pay it again.

[DESIGN-REVIEWED] 1e51676

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 23, 2026
@github-actions

github-actions Bot commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5) — ✅ PASS

Premise-level review of 1e51676c739575b42dfc3c3c9fdfc0ec74e837b8 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

One prior decision is reversed: #8350's "a duplicate comment is recoverable" — confirm you accept that a fully unreadable slot now publishes nothing, loudly.

The reversal is declared at length, and its premise checks out in-repo: the override lookup takes the first match (head -n1, first-principles-review.yml:432) and _review_contract.py:257 reads [BLOCK-MERGE] from every comment, so a duplicate really does strand the gate until a human deletes it — the pin's own rationale no longer holds. The advisory-lanes-stay-green choice is likewise grounded: pr-readiness.yml:873-894 attributes a failed Design/UX/FP lane as "(BLOCK)", so exiting non-zero there would manufacture a verdict. The one declared deferral — notice/skip writes keep the one-attempt || true shape — is named in the test as a separate defect, so nothing rides along silently.

What this change ships

Inventory (9 items) — 9 justified

Intent: a review lane that earned a verdict must actually publish it, and must never claim it published one it lost — a FIX.

  1. A lost marker write is retried (6 attempts, ~75s) and never announced as "Updated existing" — justified
  2. The two gating reads widen from ~3s to ~75s of tolerance — justified
  3. Withheld verdicts become ::warning::, unpublished verdicts ::error:: naming the head — justified
  4. New retry_comment_write primitive, byte-identical in 12 lanes, pinned by test (35 call sites, counted: 47 occurrences − 12 definitions) — justified
  5. A create now refuses an occupied or unreadable slot instead of posting anyway — reverses the fix(ci): stop fork GPT lane hiding a verdict behind an incomplete run (#8292) #8350 pin, declared, premise verified in-repo — justified
  6. A repeated PATCH re-confirms the head and stops on an accepted human override — justified
  7. Same-repo Opus/GPT lanes exit non-zero on a verdict that missed the slot; the ten shared lanes stay green with annotations — justified
  8. Override comments in the two same-repo lanes carry a new [<LANE>-OVERRIDE] <sha> line — justified (one counted consumer: the lane's confirm needle; a bare-head needle would classify a standing block as the override's own write)
  9. An earlier same-head verdict already in the slot reports as that run's write, red in the fail-closed lanes — justified

[FIRST-PRINCIPLES-REVIEWED] 1e51676

@github-actions

github-actions Bot commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ✅ human override accepted

Human judgment by @chenmingwei23 overrides the GPT 5.6 finding for 1e51676c739575b42dfc3c3c9fdfc0ec74e837b8; the recorded reason is authoritative for this commit.

[GPT-OVERRIDE] 1e51676

This comment is updated in place on each push.

The model was not re-run because an authorized human decision supersedes it.

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 1e51676c739575b42dfc3c3c9fdfc0ec74e837b8: <one-sentence reason>

@github-actions

github-actions Bot commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5 Review — ✅ no blocking findings

Reviewed 1e51676c739575b42dfc3c3c9fdfc0ec74e837b8 — this comment is updated in place on each push.

Review details

One advisory: the repeat-guard's override test is a prose substring, not the head-scoped override marker this PR adds, so any slot body containing that phrase silently cancels the retry.

FINDING — .github/workflows/claude-review.yml:820 (byte-identical in all 12 lanes; caller at :884) — case "$ov_body" in *"human override accepted"*) matches on prose with no head scoping, so when the slot holds an override written for an earlier head (verdict="✅ human override accepted" at :519 is echoed into the body at :561 and left in place until the next write) or holds a verdict body whose cat claude-review-output.md merely quotes the phrase, a single transient failure on PATCH attempt 1 sends attempt 2 into return 5 → the elif [ "$write_rc" -eq 5 ] arm prints a plain message and the step exits 0, so the lane is green while the slot keeps the other head's body and REVIEWED_STAMP_RE/sha_matches (pr_status.py:1073-1090) find no [OPUS-REVIEWED] $HEAD — a third silently-green outcome the block comment at :870-873 says does not exist → Fix: test $ov_body for [OPUS-OVERRIDE] $HEAD, the head-scoped override marker this same diff introduces at :556/:573, instead of the bare phrase.

[OPUS-REVIEWED] 1e51676

Verdict parsed from the review's SHA-scoped output markers for commit 1e51676c739575b42dfc3c3c9fdfc0ec74e837b8.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 1e51676c739575b42dfc3c3c9fdfc0ec74e837b8: <one-sentence reason>

@github-actions

github-actions Bot commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

Security Scope Review -- 🟡 CONCERNS

Which legitimate operations does 1e51676c739575b42dfc3c3c9fdfc0ec74e837b8 newly refuse? Updated in place on each push. A script-confirmed regression blocks; the model proposes candidates and never decides a refusal.

Verdict basis: the reviewer's own judgement; no confirmed regression.

Security scope review -- adjudicated candidates

  • macos-latest (platform posix): 10 classified, 0 newly refused, 10 still allowed.
  • ubuntu-latest (platform posix): 10 classified, 0 newly refused, 10 still allowed.
  • windows-latest (platform windows): 6 classified, 0 newly refused, 6 still allowed.

No candidate flipped from allowed to refused.

Scope-Verdict: CONCERNS

TIGHTENING: retry_comment_write, a single guarded publication primitive replacing the best-effort gh pr comment "$PR" --body-file ... || true / gh api --method PATCH ... || true writes in all 11 review-lane workflows -- the one threat it refuses is a non-idempotent create repeated on a lost ack planting a second marker comment in the lane's slot: "GitHub can ACCEPT it and still lose the ack, so a blind second attempt plants a SECOND marker comment. Every id lookup in these lanes selects exactly one comment, so a later human override patches only that one while the duplicate keeps its own [BLOCK-MERGE] line, and pr_status.py stays blocked until somebody finds and deletes the extra comment by hand" (.github/workflows/claude-review.yml, retry_comment_write hunk). The same primitive also refuses a blind repeated PATCH that would restore a superseded head's verdict or overwrite an accepted human override.

ADJUDICATED: 10 candidates, 0 confirmed newly-refused. No confirmed rows were handed to this run. The candidates are the exact shell shapes the tightened publish steps execute (slot pre-read, marker lookup, PATCH, head re-check, override read, stamped create, backoff), submitted so the legs can confirm the tool-call deny composite -- which this diff never touches -- does not newly refuse the lanes' own machinery. A zero here is expected and settles only the composite's half; it says nothing about the guard itself, which is the unadjudicated half below.

PLATFORM COVERAGE: any for the platform-neutral gh api/gh pr/sleep invocations, posix for every row carrying a path (/tmp/claude-summary.md), an env-var spelling ($RUNNER_TEMP), or a pipeline (printf | awk). No windows rows were written, deliberately: the tightened mechanism is bash inside runs-on: ubuntu-latest publish steps, its paths are runner-local /tmp and ${RUNNER_TEMP}, and the diff adds no assumption that any windows host ever evaluates -- there is no windows execution path for this guard, so a windows spelling would be a row nobody runs. The per-leg verdicts are the deterministic fold's to report, per leg, after this review is captured.

UNADJUDICATED: The entire tightening. The guard is retry_comment_write plus the widened six-attempt lookup/head-read budgets, duplicated across claude-review.yml, codex-review.yml, design-review.yml, first-principles-review.yml, ux-review.yml, security-scope-review.yml, and their five fork-* twins. The call path an input takes: a review lane completes → the lane's publish step runs as bash on the GitHub-hosted runner → the write to the PR's marker comment is gated by the guard's slot pre-read, head-currency re-check, override-note read, and stamp-presence check. None of that passes through the tool-call deny composite, so the classifier can issue no verdict on it, and "0 confirmed" above is not evidence about it. The candidate legitimate inputs I would have submitted, in prose:

  • A first-time publish for the current head while the slot holds a stale comment the marker lookup could not see: previously the || true create posted the fresh verdict; now return code 2 refuses and the fail-closed lanes exit 1 with no verdict in the slot until a re-run finds the lookup healthy.
  • A publish attempt whose slot pre-read fails six times during the very API-exhaustion window this PR cites (ci: a review lane that withholds its comment on a transient API read never retries, so a current-head verdict never reaches its marker #12952): previously the write was attempted anyway; now nothing is posted at all, so the outage the fix targets is also the input most likely to trip its new refusal.
  • A re-posted review-failure notice after a transient network error: an unstamped body now gets exactly one attempt, so a rejected first POST loses the notice that previously survived retries.
  • A same-head re-run asked for a fresh sample when the slot already holds that head's verdict via a failed marker lookup: return code 3 exits 1 in fail-closed lanes and the earlier sample -- including any blocking line -- stands; recovery depends on the next run's lookup succeeding and taking the PATCH arm.
  • A publish racing a head advance (return code 4) and a publish onto an accepted human override (return code 5): both are withheld writes that were previously performed.

Each of these is a workflow-internal flow, not a shell command line the composite classifies, so no verdict exists for any of them; whether each refusal is the intended trade or an over-reach is argued in the diff's own comments but confirmed by nothing this lane can run.

[SCOPE-REVIEWED] 1e51676

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 23, 2026
@chenmingwei23
chenmingwei23 force-pushed the fix/reviewer-marker-publication-retry-12952 branch from fdf39f5 to 2415ae4 Compare September 23, 2026 01:11
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 23, 2026
@chenmingwei23
chenmingwei23 force-pushed the fix/reviewer-marker-publication-retry-12952 branch from 2415ae4 to 58e2dfb Compare September 23, 2026 01:50
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 23, 2026
@chenmingwei23
chenmingwei23 force-pushed the fix/reviewer-marker-publication-retry-12952 branch from 58e2dfb to dc799ce Compare September 23, 2026 03:01
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 23, 2026
@chenmingwei23
chenmingwei23 force-pushed the fix/reviewer-marker-publication-retry-12952 branch from dc799ce to f5a1669 Compare September 23, 2026 04:06
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 23, 2026
@github-actions github-actions Bot added the readiness: action required A blocking check or review needs attention label Sep 23, 2026
@chenmingwei23
chenmingwei23 force-pushed the fix/reviewer-marker-publication-retry-12952 branch from 89321a8 to 0fe8722 Compare September 23, 2026 13:11
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 23, 2026
@chenmingwei23
chenmingwei23 force-pushed the fix/reviewer-marker-publication-retry-12952 branch from 0fe8722 to 5fe5368 Compare September 23, 2026 13:48
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 23, 2026
@chenmingwei23
chenmingwei23 force-pushed the fix/reviewer-marker-publication-retry-12952 branch from 5fe5368 to 4e51fb5 Compare September 23, 2026 14:41
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 23, 2026
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

/ai-review override gpt 1e51676: The overwrite needs two runs for the same commit both retrying inside one backoff window, the repeat already refuses on a moved head and on an accepted human override, and closing the rest needs a conditional write the comments API does not offer; the loss is a duplicate verdict sample, and re-running the lane republishes.

@github-actions

Copy link
Copy Markdown
Contributor

Human judgment recorded

@chenmingwei23 marked the gpt AI finding as false positive, not applicable, or explicitly accepted for 1e51676c739575b42dfc3c3c9fdfc0ec74e837b8.

The overwrite needs two runs for the same commit both retrying inside one backoff window, the repeat already refuses on a moved head and on an accepted human override, and closing the rest needs a conditional write the comments API does not offer; the loss is a duplicate verdict sample, and re-running the lane republishes.

This decision applies only to this commit. A new push requires a new judgment.

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving the reversal of #8350's pin, on evidence that both halves of its premise have since stopped holding.

#8350 pinned "publish anyway when the lookup fails, because a duplicate comment is recoverable and an unpublished verdict is not." On this head the verdict is no longer unrecoverable — it stays in the job log and the lane is re-runnable without a new head — and the new path never overwrites or deletes an existing verdict, so #8292's actual failure (an incomplete run clobbering a published verdict) stays closed.

The residual risk is bounded and loud: inside a >150s API-exhaustion window a finished verdict may go unpublished, which reds the two fail-closed lanes and shows up as a stale stamp rather than as a silent pass. Merge safety is unchanged — PR Readiness reads check-run conclusions, not stamp freshness.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ci: a review lane that withholds its comment on a transient API read never retries, so a current-head verdict never reaches its marker

2 participants