Repository navigation
fix(ci): retry a review lane's marker-comment write and stop claiming a lost one - #12957
Conversation
Design Review (Fable 5) — 🟡 CONCERNSDesign-level review of Design-Verdict: CONCERNS Flips the named pin Watch
Suggestions
[DESIGN-REVIEWED] 1e51676 |
First Principles Review (Fable 5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS One prior decision is reversed: #8350's "a duplicate comment is recoverable" — confirm you accept that a fully unreadable slot now publishes nothing, loudly. The reversal is declared at length, and its premise checks out in-repo: the override lookup takes the first match ( What this change shipsInventory (9 items) — 9 justifiedIntent: a review lane that earned a verdict must actually publish it, and must never claim it published one it lost — a FIX.
[FIRST-PRINCIPLES-REVIEWED] 1e51676 |
GPT 5.6 Review — ✅ human override acceptedHuman judgment by @chenmingwei23 overrides the GPT 5.6 finding for [GPT-OVERRIDE] 1e51676 This comment is updated in place on each push. The model was not re-run because an authorized human decision supersedes it. False positive or not applicable? A repository writer can comment: |
Opus 5 Review — ✅ no blocking findingsReviewed Review detailsOne advisory: the repeat-guard's override test is a prose substring, not the head-scoped override marker this PR adds, so any slot body containing that phrase silently cancels the retry. FINDING — .github/workflows/claude-review.yml:820 (byte-identical in all 12 lanes; caller at :884) — [OPUS-REVIEWED] 1e51676 Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
Security Scope Review -- 🟡 CONCERNSWhich legitimate operations does Verdict basis: the reviewer's own judgement; no confirmed regression. Security scope review -- adjudicated candidates
No candidate flipped from allowed to refused. Scope-Verdict: CONCERNS TIGHTENING: ADJUDICATED: 10 candidates, 0 confirmed newly-refused. No confirmed rows were handed to this run. The candidates are the exact shell shapes the tightened publish steps execute (slot pre-read, marker lookup, PATCH, head re-check, override read, stamped create, backoff), submitted so the legs can confirm the tool-call deny composite -- which this diff never touches -- does not newly refuse the lanes' own machinery. A zero here is expected and settles only the composite's half; it says nothing about the guard itself, which is the unadjudicated half below. PLATFORM COVERAGE: UNADJUDICATED: The entire tightening. The guard is
Each of these is a workflow-internal flow, not a shell command line the composite classifies, so no verdict exists for any of them; whether each refusal is the intended trade or an over-reach is argued in the diff's own comments but confirmed by nothing this lane can run. [SCOPE-REVIEWED] 1e51676 |
fdf39f5 to
2415ae4
Compare
2415ae4 to
58e2dfb
Compare
58e2dfb to
dc799ce
Compare
dc799ce to
f5a1669
Compare
89321a8 to
0fe8722
Compare
0fe8722 to
5fe5368
Compare
5fe5368 to
4e51fb5
Compare
|
/ai-review override gpt 1e51676: The overwrite needs two runs for the same commit both retrying inside one backoff window, the repeat already refuses on a moved head and on an accepted human override, and closing the rest needs a conditional write the comments API does not offer; the loss is a duplicate verdict sample, and re-running the lane republishes. |
Human judgment recorded@chenmingwei23 marked the gpt AI finding as false positive, not applicable, or explicitly accepted for
This decision applies only to this commit. A new push requires a new judgment. |
bolichen97
left a comment
There was a problem hiding this comment.
Approving the reversal of #8350's pin, on evidence that both halves of its premise have since stopped holding.
#8350 pinned "publish anyway when the lookup fails, because a duplicate comment is recoverable and an unpublished verdict is not." On this head the verdict is no longer unrecoverable — it stays in the job log and the lane is re-runnable without a new head — and the new path never overwrites or deletes an existing verdict, so #8292's actual failure (an incomplete run clobbering a published verdict) stays closed.
The residual risk is bounded and loud: inside a >150s API-exhaustion window a finished verdict may go unpublished, which reds the two fail-closed lanes and shows up as a stale stamp rather than as a silent pass. Merge safety is unchanged — PR Readiness reads check-run conclusions, not stamp freshness.
What is the problem?
A review lane can earn a verdict for the PR's current head and still fail to
publish it, while reporting success.
Each lane's verdict lives in one marker comment,
[<LANE>-REVIEWED] <sha>.guarded_comment_upsert()gates the write on two reads, the bot's existingcomment id and the PR's current head sha, and withholds if either cannot be
confirmed. Withholding on an unreadable head is correct: the comments API has no
If-Match, so the write is last-writer-wins and a wrong one buries a liveverdict. Three things after that point are not correct.
The two gating reads retried 3 times with 1s/2s backoff, about 3 seconds of
tolerance, against App-wide API exhaustion that lasts minutes. When they failed,
the function returned 0, the job concluded
success, and nothing retried.The write itself was attempted exactly once, under
|| true, with itsUpdated existing <lane> comment #<id>line printed unconditionally on the nextline. A lost
PATCHtherefore announced itself as a published verdict.A withheld verdict was a plain
echoin a job log, so a lane that published anda lane that withheld were indistinguishable from outside.
Why this issue matters to the user
The marker comment is the only thing a freshness verifier reads. A verdict that
does not reach it leaves the PR indistinguishable from one no lane ever
reviewed: prepare-pr's
pr_status.pyexits 20 on a PR whose check-runs,PR Readinessstatus andmergeablestate all pass, and that refusal is correct,because the markers really are stale.
The only recovery was manufacturing a new head. That re-rolls every
non-deterministic review lane and discards the verdicts already earned, so the
cheapest way out of a lost write cost more than the review it lost. A reviewer
lane that fails to publish is worse than one that fails red: the gate stays
green, the marker stays old, and every prepare-pr and babysit loop reads
BLOCKED with nothing to act on.
How our fix solves it
The symptom is a stale marker. The marker is stale because the write never
landed. The write never landed because it got one attempt and no verification,
and the reads gating it gave up in about 3 seconds. So the fix is at the
publication chain, and nothing in
pr_status.pychanges: a stale marker isstill read as stale.
retry_comment_write()is one publication primitive. Six attempts, 5s linearbackoff, about 75s, and it returns the outcome instead of swallowing it. The
caller owns the wording, because only the caller knows which body it was
publishing. On success it prints what it printed before; on an exhausted budget
it emits an
::error::annotation naming the head whose verdict is unpublished.Both gating reads move to the same budget, since one exhaustion window fails
reads and writes alike, and the two withhold notices become
::warning::annotations so a withheld verdict is distinguishable from a published one.
The two write kinds are not equally safe to repeat
A
PATCHis idempotent: same URL, same body, so repeating it changes nothing.A create is a
POSTand is not. GitHub can accept it and still lose the ack, soa blind second attempt plants a second marker comment. Every id lookup in
these lanes selects exactly one comment, so a later human override patches only
that one while the duplicate keeps its own
[BLOCK-MERGE]line, andpr_status.pystays blocked until somebody finds and deletes the extra commentby hand. That is the same gate-stranding this change exists to remove, reached
from the other side.
So a create may only repeat once it can RECOGNISE its own work, and the needle
is the current head's stamp. Only a run for this head writes
<stamp> <sha>, soa comment left by an older head cannot satisfy it. Marker presence cannot be the
needle: every body these lanes write starts with the lane marker, so a stale
comment would answer "landed" for a verdict that was never published, and the
retry that would have published it would be abandoned, which is this same defect
class arriving from a third direction.
A
PATCHpasses"" ""and repeats freely. A create passes the current head'sstamp and the body it is posting, and before each repeat the primitive re-queries
for that stamp. A body carrying no such stamp (a review-failure notice) has
nothing a later query can recognise, so it gets one attempt: for it a lost ack
and a rejection are indistinguishable, and repeating could duplicate it
undetectably.
An unreadable confirmation stops the create rather than repeating it, because
the two outcomes are not symmetric. A duplicate strands the gate with no
automatic recovery; an unpublished verdict is annotated, leaves the gate honestly
red, and a re-run fixes it. That is the same rule the head-confirmation read
already follows: the destructive half must not proceed on an unknown.
An unpublishable verdict is an infrastructure failure, not a BLOCK verdict, so
the advisory lanes still exit 0 and the annotation carries the signal. The one
exception is
codex-review.yml's create, which has always been bare and failedthe step; the retry is added there without softening that, so only a genuinely
unpublishable verdict still exits non-zero.
Which lanes, and why that set
The set is every lane that publishes a verdict, which is wider than the
three lanes that went stale:
guarded_comment_upsertdesign-review,first-principles-review,security-scope-review,ux-review, and the sixfork-*lanesclaude-review.yml(Opus 5),codex-review.yml(GPT 5.6)Design and First Principles went stale through the shared function's withhold
path. Opus went stale through
claude-review.yml, which has no guard at all; itis the lane whose log said it updated a comment that was never updated. GPT
carried the identical suppressed-write shape and escaped only because
codex-review.ymlhas an extraeditedtrigger that an unrelated body editre-fired. Fixing the shared function alone would have left the same defect in
the one lane whose recovery path everything else depends on.
retry_comment_writeis byte-identical in all twelve, pinned by a test, the wayguarded_comment_upsertalready is in ten.Two questions, two needles
A create is only safe once the slot is known empty, and only safe to repeat once
the previous attempt is known not to have landed. Those are different questions
and they need different needles.
The slot is every comment starting with the lane marker, whatever head it
carries, because that is what each lane's own id lookup selects. A caller reaches
a create whenever its own lookup could not answer, so an occupant it could not see
is exactly when a POST puts a second comment in a slot that holds one. The
occupancy read therefore matches
startswithon the lane marker, before the firstwrite:
Landing is a different question, answered after the write by
containson thecurrent head's stamp, which only a run for this head writes. The lane marker
cannot answer it: an older head's comment carries the marker too, so matching on
it would report a verdict published that never was. Since the slot was confirmed
empty first, a stamp found afterwards is this run's own landed write.
A PATCH names a known id and is idempotent, so it passes an empty slot and an
empty stamp and takes neither check.
An occupied slot is refused, and a stranded verdict goes red
Published::error::, and see the lane table below::error::, same::error::, sameOne comment answers for this lane, so every alternative to refusing loses
something no run gets back. A second comment beside the occupant is the one a
later human override cannot reach. Overwriting the occupant can discard a verdict
this run has no standing to judge: a concurrent run's for the same head, or the
CURRENT head's when this run's own head is already superseded.
Refusing is only half an answer, though, because a lane that publishes nothing and
says nothing hides exactly the stranding this change exists to fix -- the PR reads
as unreviewed for its own head with every check green and nobody told. So every
such case emits an
::error::annotation naming the head whose verdict is not onthe PR. The one occupant that costs the PR nothing is the slot already carrying a
verdict for this head: the stamp readers find the head they are looking for, so
status 3 keeps that case distinct and no caller reports another run's write as its
own.
Whether that also fails the lane depends on which lane it is
Design Review,UX Review,First Principles Review(BLOCK), a judged-wrong designOpus 5 Review,GPT 5.6 Reviewpr-readiness.ymlnames exactly those first three lanes for the(BLOCK)(failure)is the only attribution true of both a real verdict and a run that/ai-review override".The two same-repo reviewer lanes are fail-CLOSED, and their gate reads the captured
review text rather than the PR. So a verdict that missed the slot leaves the check
green while the PR presents an older head's verdict -- and
codex-review.ymlalreadyexits non-zero in one arm of that same
if/elifchain, under a comment saying anunpublishable verdict fails the step rather than passing quietly. Both lanes now
follow one rule: an outcome that leaves this head's verdict out of the slot exits
non-zero, which is four arms each. Status 4 stays green, because the run for the
current head publishes the verdict that counts.
Status 3 divides the two classes as sharply as a failed write does. An earlier run
for this SAME head already holding the slot costs an advisory lane nothing it can
act on. A fail-closed lane loses the fresh sample the re-run was asked for, and the
earlier verdict keeps whatever it said, a blocking line included -- so that verdict
goes on gating the PR while the lane reports green. The two fail-closed lanes
therefore name the occupant and fail; the ten shared lanes keep it green. Nothing
writes into the slot in either case: replacing what another run put there needs
standing the primitive does not have, which is why #13100 owns that question.
This happened in production on this PR while it was being reviewed, which is how the
distinction got measured rather than argued. On head
14350d53e7the Opus laneproduced a clean verdict, then lost every one of the six attempts at its comment
lookup AND every one of the six at the slot read to an API window, published
nothing, and concluded
success-- leaving the slot holding a verdict for a head twopushes back. Nothing in CI caught it: the required readiness status reads check-run
conclusions and carries no stamp-freshness read at all, so the reader that sees a
head with no verdict is prepare-pr's own marker evaluation, which a human runs.
That is also why the shared lanes' annotation is not a lesser channel dressed up as
a choice: for an advisory lane it is the only honest one available.
The retry re-reads the whole slot rather than only bodies carrying this run's
needle. The slot was confirmed empty before the first write, so anything there
during the backoff arrived meanwhile: this run's own write whose ack was lost, or
a run for a different head publishing while this one waited. Reading only this
run's needle made the second invisible, and the retry then posted beside it.
A repeat re-confirms the head, and what that still cannot cover
The slot re-read protects a create. A replace has nothing equivalent: the caller
names the comment id and the primitive never reads that comment's body, so there
is no shape to compare and the repeat is blind. If the PR moves to a newer head
between two attempts, the run for that head owns the comment, and the second
attempt puts a superseded body back over a current verdict.
The head is the standing a replace rests on, so it is re-confirmed before every
repeat, on the same terms as the write it gates. A head that has moved stops the
loop and answers 4, which is not an error: the run for the current head publishes
the verdict that counts, so nothing is missing from the PR. An unreadable head
stops the loop too, because repeating is the destructive half and must not proceed
on an unknown -- what stopping costs is the recoverable side, an unpublished
verdict that is annotated and that a re-run publishes. The order inside the loop
matters and is pinned: the landing read runs first, so a write whose ack was lost
is still reported as landed rather than as a superseded no-write.
Residual, stated rather than implied. Two runs on the SAME head are alike to a
head check, so a same-head run that replaces the comment between two attempts has
its body overwritten by the repeat. Both bodies are verdicts for that head, so
every stamp reader finds the head it looks for and no gate is stranded; what is
lost is which sample is displayed. Telling the two apart needs a body pre-image or
a conditional write, and this API offers neither an
If-Matchnor an edit history,so it cannot be closed inside this chain.
One same-head body is an exception, because losing it is not a display question: an
accepted human override REPLACES this slot's comment, and a verdict restored on top
of it puts back the block the override cleared, which no later run undoes. So a
repeat re-reads the comment it is about to write and stops when that comment says a
human overrode the lane. That is one bounded read of one id, not a compare-and-set:
it keeps no pre-image and recognises nothing but the override note, so the ordinary
same-head residual above stands unchanged. An unreadable answer stops the repeat on
the same terms as an unreadable head.
An accepted override carries its own marker
One body in this slot is authoritative for the head and carries no reviewer stamp:
the human override. Named by the stamp, it was classified a notice and left out,
so an accepted override left a standing block in the slot with nothing later to
recover it. Named by the bare head, it matched the blocking verdict it was posted
to clear, and the slot read then read that verdict as this run's own write and
reported the override as already published. So an override body emits, and is
matched by, a marker only an override carries:
[OPUS-OVERRIDE] <sha>and[GPT-OVERRIDE] <sha>, on the same head-scoped shape as the verdict stamps.The lookup that gates the write is retried too
A caller may only replace the lane's comment if its own lookup succeeded -- that
read is what tells it the comment it found is the one to update. Two lanes looked
theirs up with a single attempt that swallowed its own error, so an ordinary flake
answered "no comment", routed a live verdict into the create path, and met the
refusal above. The result was a lane publishing nothing for a head the PR then
read as unreviewed: the same stranding by a different door, and worse than the
duplicate it replaced, because a duplicate at least left a verdict on the PR.
Those lookups now get the widened budget every other gating read in this chain
has, and they report their failure instead of hiding it.
One contradiction resolved, not a new trade
The withhold branch already refused to create when the lookup errored, on the
grounds that "a marker planted is not undone by a later run". Ten lines below, the
verdict branch created anyway, on the grounds that "a duplicate comment is
recoverable and an unposted verdict is not". Both cannot hold, and the second is
the false one: clearing a duplicate takes a person finding and deleting a comment,
while a withheld verdict is published by re-running the lane. The same unknown now
gets the same answer in both branches, and the prose asserting otherwise is
corrected in the same change.
Where the occupancy read answers cleanly and the slot is empty, a completed
verdict publishes exactly as before even though the caller's lookup failed, which
test_completed_verdict_creates_once_a_duplicate_is_ruled_outpins.One thing needs a human, not a lane.
test_completed_verdict_creates_when_the_lookup_failswas a named pin, and this replaces it. The Design lane's advisory CONCERNS asks the
merger to ratify that flip rather than letting the change carry it silently, which
is right, so it is stated here: merging this accepts that a review lane withholds a
verdict it cannot publish safely, instead of publishing one that may strand the
merge gate. First Principles independently confirmed the unrecoverability premise
against
pr_status.py, which blocks on ANY bot comment carrying[BLOCK-MERGE]forthe current head while every lane's id lookup selects exactly one comment. Security
Scope classified 23 candidates across three platforms and confirmed 0 newly
refused, naming the same window as the one behavioural loss and
gh run rerunasits recovery.
The same defect, reported separately as #13011
#13011 reports
Opus 5 Reviewconcludingsuccesswhile the PR carries no[OPUS-REVIEWED]stamp for its head, sopr_status.pyanswersstale reviewer stamp(s)on four of four sampled PRs. That lane's publicationpath on
mainis a suppressed write followed by an unconditional claim:So the PATCH's rejection is discarded, the log says the update happened, the
comment keeps an older head's stamp, and the lane stays green. That is this
change's own root cause, on the one lane that can prove the verdict existed:
Gate on Opus 5 reviewrunsalways()and fails closed unless[OPUS-REVIEWED] $HEADis in the output, so asuccessconclusion on an ordinaryPR means the review completed for that head and the write is where the stamp was
lost. (Two conditions pass that gate without a verdict, a recorded human override
and a dependabot PR, and neither is present on the four sampled.)
Both of that lane's writes now go through
retry_comment_write, so a rejection isretried on the widened budget and, if it persists, reported as
::error::Opus 5 ... could not publish it; the success line prints only for aconfirmed write. What this does not do is guarantee the write lands. A rejection
that is deterministic rather than transient -- an oversized body is the candidate
that would explain four of four -- survives every retry. It converts a silent
stale stamp into a named failure carrying GitHub's own reason, which is what
telling those two cases apart needs, so #13011 stays open for that measurement
rather than being closed here.
Deliberately out of scope
Some of these steps have other marker writes, for human-override notes and for
skip and no-contract notices, that still carry the one-attempt
|| trueshape.They are a separate defect with the same symptom: they sit before their step's
early
exit 0, above the primitive's definition, so covering them meanshoisting the definition through twelve heterogeneous step preambles. That belongs
in its own change rather than folded in here, and the test that pins this
contract says so where it scopes itself to the verdict writes.
A re-fire entry that needs no new head, a
workflow_dispatchor a sweep shapedlike
pr-readiness-sweep.yml, is also not here. It is a recovery mechanism forverdicts already lost, not a fix for losing them, and it is a new workflow rather
than a change to this chain.
Replacing an occupied slot is not here either, and #13100 tracks it. Deciding that
this run's body should displace a comment another run put there needs standing this
primitive does not have -- it knows the id, not what the occupant says or which run
wrote it -- so the decision belongs to a caller that can establish it. Refusing is
the conservative half of that trade and is loud: the verdict is annotated and a
re-run publishes it.
What tests we did
test/test_ai_review_workflows.py, 945 passed, run with an explicit-n0.31 red against the head before this one: the override guard on an occupied target and on an unreadable one, the per-repeat target read inside the lost-write test, across ten lanes each, plus the primitive's identity pin. Four uniform mutations across all twelve lanes were each caught with none uncaught: the guard removed, an unreadable target treated as safe, the guard reading the target but never recognising an override, and the override outcome reported as this run's own publication.
21 red against the head before that one: the lane-class contract, plus the two behavioural tests that pin how an advisory lane reports an earlier same-head run's occupancy, across ten lanes each. Four uniform mutations were each caught with none uncaught: the discarded fresh sample going quiet in either fail-closed lane, a superseded run treated as a loss, and the same failure pushed into an advisory lane whose failure readiness reads as a judged BLOCK.
One red against the head before that one: the contract that pins which lane class exits on a verdict that missed the slot, which also asserts the shared function carries no
exitin any arm. Four uniform mutations were each caught with none uncaught: the occupied-slot arm going quiet in a fail-closed lane, the replace arm going quiet there, a superseded run treated as a loss, and anexit 1added to a shared lane.62 red against the head before that one, over eight contracts: the retry's head re-confirmation on a moved head and on an unreadable one, its per-repeat read count, the three cases that must leave an advisory lane green rather than fail it, the override body's own marker, and the primitive's identity pin. Four uniform mutations across all twelve lanes were each caught with none uncaught: the head never re-confirmed before a repeat, an unreadable head treated as still current, the re-confirmation gated on the needle so only the create is covered, and the re-confirmation moved above the landing read so a landed write reports as a superseded no-write.
Refusing an occupied slot: 31 red against the head that wrote into one, being
test_an_existing_current_head_comment_is_never_duplicated,test_a_superseded_head_comment_is_refused_tooandtest_a_concurrent_runs_comment_is_not_reported_as_this_runs_writeacross ten lanes each, plus the primitive's identity pin. Four uniform mutations -- the pre-write occupant answering success, the retry-time occupant answering success, the occupant check never firing, and an occupied slot overwritten again -- were each caught, none uncaught.The needle's ordering and the retry's slot read: 2 red against the head that had neither, being the call-site contract's order assertion and the primitive's identity pin. Three uniform mutations -- the needle computed above the classification again, the retry read matching only this run's needle, and any occupant reported as this run's own landed write -- were each caught, none uncaught.
One answer for an occupied slot: 12 red against the head that had two, being
test_a_stale_head_comment_is_replaced_not_refusedacross ten lanes, the primitive's identity pin, and the call-site contract that pins the override needle. Four uniform mutations -- an occupant never written, a failed update reporting success, the update turned back into a create, and the override body stripped of its head-scoped needle -- were each caught, none uncaught.Writing into an occupied slot rather than stopping: 31 red against the head that stopped, being
test_an_existing_current_head_comment_is_never_duplicated,test_a_concurrent_runs_comment_is_not_reported_as_this_runs_writeand the newtest_a_slot_update_that_cannot_land_is_not_reported_as_publishedacross ten lanes each, plus the primitive's identity pin. Three uniform mutations across all twelve lanes -- the occupant preserved again, a failed update reporting success, and the update turned back into a create -- were each caught, none uncaught.The occupant-is-two-answers change: 31 red against the head that carried the single answer, being
test_a_stale_head_comment_also_occupies_the_slot,test_an_existing_current_head_comment_is_never_duplicatedandtest_a_concurrent_runs_comment_is_not_reported_as_this_runs_writeacross ten lanes each, plus the primitive's identity pin. Three uniform mutations across all twelve lanes -- the stale branch returning success, the slot read dropping the stamp field, and the no-stamp branch moved back above the slot question -- were each caught, none uncaught.An earlier head carried four reds this change did not cause, all owned by
mainand tracked as #13005:
Backend Tests (3.12, 7)andBackend Tests (Windows) (7)on
test/test_crew_log_exc_info_sites.py, theCoverage Gatedownstream of them,and
Frontend Lint & Type Checkon a restyle ratchet this diff has no frontendfile to affect. Their fixes are #13007 and #13008, both merged, and this branch is
rebased onto a
mainthat has each of them as an ancestor, so those lanes nowmeasure this diff.
Twelve reds on that same head were one API rate-limit window, not verdicts:
API rate limit exceeded for installationat 05:31:43Z, with every reviewer lanefailing closed on its own inability to read this PR's comments. The rebase re-rolls
them rather than a re-run spending into an unrecovered window.
Red-on-base first. The new and updated tests against unmodified base workflows:
54 failed, 58 passed. Every contract is red without the fix, including both
lost-write tests across 10 lanes each, the no-unconditional-success-claim test
across all 12, the primitive identity test, and the two widened read-budget tests
across 10 lanes each.
Each reviewer finding was proved the same way, against the head that carried it.
The non-idempotent-create finding: 12 failed, 111 passed, the failures being
test_a_create_whose_ack_was_lost_is_not_posted_againacross 10 lanes plus twocontract tests. The stale-marker confirmation finding: 12 failed, 121 passed,
the failures being
test_a_stale_marker_never_confirms_an_unpublished_verdictacross 10 lanes plus the same two contract tests. The stale-marker test drives
the exact interleaving the reviewer named:
STUB_COMMENT_FAIL_FIRSTfails onlythe first N comment lookups, so the id lookup errors, the step takes the create
path, the POST fails, and the confirmation query then succeeds with an older
head's comment on the PR.
The occupancy-needle finding: 42 failed against the head that carried it,
being
test_an_existing_current_head_comment_is_never_duplicated,test_a_stale_head_comment_also_occupies_the_slot,test_a_concurrent_runs_comment_is_not_reported_as_this_runs_write,test_an_unreadable_duplicate_check_refuses_the_writeand the updated lost-acktest across 10 lanes each, plus the identity pin and the call-site contract. The
stale-head test is the crisp one: a comment for an OLDER head on the PR,
STUB_COMMENT_FAIL_FIRST=6so the caller's lookup errors on all six attempts andit takes the create path, and nothing failing the write, so an unguarded POST
succeeds and sits a second comment beside it. It asserts the create file does not
exist at all. Two tests hold the ordinary path working:
test_a_first_write_still_happens_when_the_slot_is_provably_emptyandtest_completed_verdict_creates_once_a_duplicate_is_ruled_out.The behavioural tests run each lane's real step bash with a stubbed
gh,injecting the failure.
STUB_PATCH_FAILandSTUB_CREATE_FAILfail every writeattempt and record the count, and record no accepted body, so a test cannot
assert on a body the API never took.
STUB_CREATE_LANDSemulates the lost-ackpartial failure: GitHub accepts the POST and the CLI still reports failure, with
the body written into the finder fixture so the step's own confirmation query
sees it through real
jq. The tests check the lane does not printUpdated existing/Created new, emits an::error::naming the head, postsexactly once when an ack is lost, and still exits 0. A
sleepshim on the stubPATH records the backoff schedule (
5 10 15 20 25) instead of waiting it out, sothe lanes keep their real production budget with no test-only knob in the
workflow.
Mutation proof, each mutation applied uniformly across the lanes so the
byte-identity pin stays satisfied and only behaviour breaks. Four on the retry
budget and the success claim: budget shrunk to one attempt, a write failure
announced as a published update, the primitive always reporting success, and every
lane reverted to the bare suppressed write. Four on the occupancy check: the
occupancy read keyed on the head stamp instead of the lane marker, an occupied slot
ignored, an unreadable answer treated as an empty slot, and the check skipped
entirely. All eight caught by the behavioural tests, zero uncaught.
Gates:
scripts/check_comment_history.pyself-test then the gate itself(passed), prepare-pr
preflight.py(READY),push_guard.py --base main --max-ahead 2(SAFE TO PUSH), and all twelve workflows parse as YAML.Pattern harvest
Rule candidate: semgrep
Pattern: a mutating API call whose failure is suppressed with
|| true,immediately followed by an unconditional success log line. The suppression and
the claim are individually defensible and only wrong together, which is why
review kept reading past them. The shape is mechanically detectable in shell:
a command ending
|| truewhose next statement is anechoasserting the writehappened. This PR removes five instances of it; the same shape remains in the
override-note and skip-notice writes named under "Deliberately out of scope",
which is evidence the class is worth a rule rather than twelve hand fixes.
Second rule candidate: review-prompt
Pattern: a retry loop wrapped around a non-idempotent request. Adding a retry is
usually read as strictly safer, so the review question "is this request safe to
repeat, and how would a repeat be detected" is not asked. It should be asked
whenever a retry is introduced around a
POST, since the failure it creates(a duplicate that strands a gate) can be worse than the one it fixes. Two rounds
of this PR's own review were spent on that question: first that the POST could
duplicate, then that the confirmation added to prevent it could false-positive on
a stale comment. A reviewer caught both, which is the argument for asking the
question in the prompt rather than per PR.
Any other suggestions on the work
claude-review.ymlstill has no head-confirmation guard, so it can publish averdict for a superseded head, which is the loss the shared guard exists to
prevent arriving from the other direction. Converging it onto
guarded_comment_upsertchanges that lane's withhold semantics and deserves itsown review; this change only makes its write reliable and its log honest.
The retry budget is calibrated to one observed window. If a future exhaustion
outlasts 75s the annotation will now say so plainly, which is the input needed to
retune it.
Closes #12952