Repository navigation
fix(ci): re-run an advisory review lane whose withheld verdict never published - #17698
Conversation
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Check one real sweep tick (same-repo and fork) where a GITHUB_TOKEN [DESIGN-REVIEWED] f086c7d Description read[DESCRIPTION-READ] e0e4e8c09068201fe8f13315225da1f5f87be0f9d6f8617f84949724c19e5f12 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5) — 🟡 CONCERNSPremise-level review of First-Principles-Verdict: CONCERNS The new script re-implements the override handler's lane-run lookup, and the fork path has already drifted from it: it drops the handler's per-PR row binding. Not justified as shipped
What this change shipsInventory (6 items) — 5 justifiedIntent: a PR whose advisory verdict was computed but withheld gets that verdict published without a human re-running the lane. FIX (#13100).
[FIRST-PRINCIPLES-REVIEWED] f086c7d Description read[DESCRIPTION-READ] d204a97ea2091a7adb3c99d2aa020d88465f2175c4d2797219c630ce0d593435 That sha256 names the PR description this verdict read. Recompute it with |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
buluoray
left a comment
There was a problem hiding this comment.
Review — request changes (one blocking finding)
This PR adds a recovery for a withheld advisory verdict: PR Readiness stamps [verdict-unpublished], and the self-heal sweep re-runs the owing Design/UX/First Principles lane through republish_withheld_verdict.py. The scope, fork handling and token use look right. The blocking issue is that the re-run has no attempt limit.
Verified
- Blocking lanes are never re-run.
owed_lanesreadspr_status.py --disposition-gate'sunpublishedlist, which is restricted toWHOLE_DESIGN_LANES(pr_status.py:2053-2063), and then filters toLANES(the three advisory lanes). Opus, GPT and Security Scope cannot reach_rerun. - It does not overwrite a newer verdict. The re-run lane publishes through its existing guarded upsert. That upsert refuses a superseded-head body (
design-review.ymlguarded_comment_upsert) and re-confirms the head before any repeat write (confirm_head, rc 4). If the PR moves between the sweep's read and the re-run, the lane stands down. A re-run requested while the run is still queued or in progress is rejected by the API and logged, so it is not doubled. - Token permissions and fork safety. The workflow
permissions:are unchanged (actions: writealready existed for the dispatch). The scripts run from a sparse checkout of the default branch withpersist-credentials: false, and no fork code runs. A fork Stage-2 run id read from the check-run marker is re-checked against.github/workflows/fork-*-review.ymlbefore re-running. The re-run reuses the same privileged path the override handler already uses, so there is no new trust surface. - The sparse checkout covers everything
pr_status.pyloads (_review_contract.py,green_age.py).
Mutations (run locally)
| Guard removed | Test that failed |
|---|---|
name in LANES filter in owed_lanes |
test_owed_lanes_drops_names_this_script_cannot_rerun |
fork run path verification |
test_fork_lane_marker_pointing_at_the_wrong_workflow_is_refused |
gate ok fail-closed check |
test_owed_lanes_fails_closed_when_the_gate_is_not_ok |
same-repo head_repository.full_name binding |
none: only head_branch is exercised |
Tests run locally: test/test_republish_withheld_verdict.py (9 passed). test/test_pr_readiness_sweep.py could not run here: it times out per test in this sandbox, identically on origin/main, so I only checked the sweep shell changes by reading the source.
Blocking
The automatic re-run has no attempt bound. The recovery ends only when a re-run publishes. It is fail-open on cost if the cause of the withhold persists. Withhold causes include a slot or head read that stays unreadable after 6 attempts, which is exactly the API-exhaustion case, or a PATCH that keeps failing. In that case the loop is:
- The re-run withholds again.
- Readiness republishes the pending with the token.
- After
STALE_MINUTES(15) the sweep re-runs the lane again.
That is a full paid model review roughly every 15-20 minutes, per lane, per PR. The sweep places it deliberately outside MAX_DISPATCH, so the only real bound is GitHub's platform limit of 50 re-runs per workflow run. GitHub's changelog says an attempt past that limit produces a failed check suite. I have not verified whether that would surface as a failed lane, and therefore a BLOCK, in readiness. The shared-head fork mis-binding below makes the same loop reachable with no infrastructure fault at all.
Fix: have _same_repo_run_id / _fork_run_id also return the run's run_attempt. It is already in the runs-list payload, and the fork path already reads actions/runs/{id}. Skip the re-run when run_attempt >= N (for example 3) and log "re-run this lane manually". Add a test that pins the cap and fails when it is removed.
Non-blocking follow-ups
pr-readiness-sweep.yml:519: the old short-circuit avoided the status read while the pending was younger thanSTALE_MINUTES. Movereadiness_descriptionback insideif [ "$age" -ge "$stale_seconds" ]; both consumers are age-gated anyway. This removes an extra read per tick on every young pending in the shared REST pool. (Opus finding, confirmed.)_fork_run_idpicks the newest check-run by name with noexternal_id == "<lane>-pr-$PR-<trigger>"binding, unlikeai-review-human-override.ymlbind_fork_trigger. On a shared fork head it can re-run a sibling PR's lane. Bind it the same way, or have one implementation call the other. (First Principles finding, confirmed.)- Add a test that fails when the same-repo
head_repository.full_namebinding is removed. - The docstring's root-cause sentence describes only the create-path refusal. The replace path does overwrite an older head's comment, so say when a re-run actually helps.
republish()'sis_fork/head_repo/head_refparameters are used only by tests; consider stubbing_resolve_pr_metainstead._gh_jsonandowed_lanescallsubprocess.runwith no timeout. One hungghcall can hold the sweep's concurrency group until the 15-minute job timeout.
Upgrade impact
None for users: this is CI-only with no config or state. The readiness description gains an optional [verdict-unpublished] prefix, placed after [read-failed] so the existing prefix match still holds.
|
Thanks for the thorough review — the verification of the scope, fork handling and non-overwrite guarantees matches the intent. Addressing each point against new head Blocking: the re-run has no attempt bound — fixedAdded a bounded re-run in
Tests pin the cap and fail if it is removed: Mutations table — the missing same-repo
|
|
Follow-up filed for the five deferred non-blocking items (1, 2, 4, 5, 6 above): #17788. |
UX Review (Opus 5.5) — ⏭️ skippedRevision |
…published (#13100) A whole-design review lane -- Design, UX, First Principles -- can compute a verdict for the current head and fail to publish it, because the lane's shared comment-write primitive refuses to overwrite a comment slot an earlier head still occupies. The lane completes `success` and reports honestly that it published nothing (#12957), the verdict is retained as a workflow artifact (#13911), and PR Readiness scores the lane as a named pending rather than passed (#13419). What was missing is the recovery: nothing re-ran the lane, so every consumer that keys on the head (`pr_status.py`'s stale-stamp report) read the PR as unreviewed for its own head until a human noticed the warning in a job log. This adds the sweep-triggered recovery the issue asks for, in the shape of `pr-readiness-sweep.yml`'s existing dropped-signal self-heal and reusing #13419's single predicate, rather than the two rejected alternatives (replaying the retained artifact reopens the authority question the write primitive's refusal exists to answer; a caller-supplied replace permission is a trust-boundary change to the shared primitive). What changed: - pr-readiness.yml stamps a stable `[verdict-unpublished]` token first on the pending status description whenever an advisory lane is scored `(verdict not published, re-run this lane)`, mirroring the `[read-failed]` token so a 140-char truncation cannot strip it. - pr-readiness-sweep.yml, on a stale pending carrying that token, runs the new .github/scripts/republish_withheld_verdict.py instead of re-firing the aggregator (which would only re-derive the same named pending). The script asks pr_status.py's `--disposition-gate` which advisory lanes are owed -- the one `evaluate_reviewer_markers` predicate both gates answer from -- locates each owing lane's run (same-repo by head; fork via its `<!-- ai-review-fork-lane run=<id> -->` check-run marker, as the human-override handler does) and `gh run rerun`s it. The re-run recomputes and republishes through the lanes' own guarded upsert, whose authority rule is unchanged, so the recovery adds no new trust surface. Scoped to the three advisory lanes by construction: the required reviewer lanes (Opus, GPT, Security Scope) fail CLOSED on a withheld verdict and are recovered by their own check-run finalize, never read as a stale `success`. Self-terminating: once a re-run republishes, the lane leaves the gate's `unpublished` set, the token clears, and the next sweep does nothing; a re-run in flight carries an `in_progress` check-run readiness scores `(not started)`, so the dispatch-to-republish window re-fires nothing. Tested: unit tests for the republisher (lane location same-repo + fork, the fork-marker workflow verification, fail-closed on an unreadable gate, the lane table pinned against the override handler); behavioural sweep tests that a `[verdict-unpublished]` pending re-runs the lane and does not dispatch the aggregator; token-agreement and per-site stamp pins across the two workflows; updated the sparse-checkout pin. Not run on a live forge. Fixes #13100
review) A persistent withhold cause (an unreadable head/slot read, a PATCH that keeps failing) would make the sweep re-run a paid advisory review lane every STALE_MINUTES forever, since the recovery is placed outside MAX_DISPATCH on purpose. _same_repo_run_id and _fork_run_id now also return the run's run_attempt -- already in the runs-list payload, and in the actions/runs/{id} read the fork path already makes -- and the sweep stops re-running a lane once its attempt count reaches MAX_RERUN_ATTEMPTS (3), logging a manual-re-run request instead. Also adds the mutation test the review named for the same-repo head_repository.full_name binding, which no prior test exercised in isolation.
71ad570 to
f086c7d
Compare
|
Reviewed again at the new head, still not approving (crew-pr-reviewer). It edits the readiness workflows, and @buluoray's change request is still open. Eight checks are red on this head, including Sensitive Change Review. |
buluoray
left a comment
There was a problem hiding this comment.
Approve. The blocking finding from the previous round is fixed at f086c7d241.
Prior blocking finding: FIXED. republish() now reads run_attempt for the located run and stops at MAX_RERUN_ATTEMPTS = 3 (.github/scripts/republish_withheld_verdict.py:105-115, :359-371). It prints a manual re-run request instead of calling _rerun. The attempt comes from payloads the script already fetches: the runs-list entry on the same-repo path (:197-215) and the actions/runs/{id} verification read on the fork path (:270-276), so the cap adds no REST call. A missing or malformed run_attempt defaults to 1. That means the run gets re-run, the same behaviour as before this PR, but it is limited to the rare case where GitHub omits the field.
What I checked
- The targeted tests pass at the head.
- I turned off the cap check, and both
test_same_repo_run_at_the_attempt_cap_is_not_rerunandtest_fork_run_at_the_attempt_cap_is_not_rerunwent red. - I forced the same-repo attempt read to 1, and the same-repo cap test went red. Forcing the fork attempt read to 1 turned the fork cap test red.
- One attempt below the cap still re-runs (
test_same_repo_run_below_the_cap_is_still_rerun). - I have not exercised this against live Actions. The cap depends on
gh run rerunincrementingrun_attempton the same run id, which is GitHub's documented behaviour.
Non-blocking follow-ups
docs/ci/ci-and-reviews.md(the "A withheld advisory verdict is re-run and republished" section) and the sweep's inline comment at.github/workflows/pr-readiness-sweep.yml:520-540still describe the recovery as only "self-terminating". Add a sentence on the 3-attempt cap and the manual re-run it asks for.- Carried over: a young pending still costs an extra REST read per tick.
- Carried over: the fork run lookup verifies the workflow path but does not bind the check-run's
external_id. - The manual re-run request only appears in the sweep job log. Consider putting it somewhere a human will see, such as the readiness description or a PR comment, so a stuck lane does not stay quietly pending.
Upgrade impact: none. The script and sweep step are new in this PR, and the cap only narrows them. Audience fit: CI-only. The default path (a verdict that publishes normally) never reaches this code.
Why this change
The issue was filed by the maintainer (
chenmingwei23) after review of #12952/#12957, which deliberately left the recovery for a withheld advisory verdict unimplemented. It affects any PR whose Design/UX/First Principles verdict is computed but lands in an already-occupied comment slot — the PR then reads as unreviewed for its own head and sits at a pending readiness state until a human notices a job-log warning and re-runs the lane by hand. This is CI-infrastructure only: it changes no user-facing default, config, UI or output, and adds no setting; it is a maintainer/reviewer-facing reliability fix for a common steady-state arrangement (a slot occupied by an earlier head's verdict), not a niche need.Problem / Motivation
Goal: A whole-design review lane that computes a verdict for the current head but cannot publish it is automatically re-run so its verdict reaches the pull request, instead of the PR reading as unreviewed until a human intervenes.
A whole-design advisory lane (Design, UX, First Principles) can finish
success, compute a verdict for the current head, and still fail to publish it: the shared comment-write primitive refuses to overwrite a comment slot an earlier head still occupies. The lane reports honestly that it published nothing, the verdict is retained as a workflow artifact, and PR Readiness scores the lane as a named pending —(verdict not published, re-run this lane)— rather than passed. Nothing then re-runs the lane, so the pull request stays stuck as unreviewed for its own head with no automatic recovery.Why it matters
A PR whose advisory verdict is withheld sits in a pending readiness state indefinitely. The only recovery today is a human noticing and manually re-running the lane. On a busy queue that is slow and easy to miss, and it is exactly the kind of self-healing the readiness sweep already provides for the analogous dropped-signal case — so the gap is both user-visible (a blocked PR) and inconsistent with the existing recovery design.
Not a goal
success.What changed (motivation → approach → change)
success, its verdict never reaches the comment slot, and readiness parks the PR at a named pending forever.pr-readiness-sweep.yml) re-fires the readiness aggregator, but a recompute only re-derives the same named pending — nothing re-runs the owing lane, which is the only action that republishes the verdict.actions: write, 5-minute, self-terminating re-fire engine. Give the withheld-verdict case a stable signal and let the sweep re-run the lane rather than the aggregator, reusing the one predicate that already decides which lane is owed.pr-readiness.ymlstamps a stable leading token[verdict-unpublished]on the pending status description when an advisory lane owes the head a verdict — mirroring the existing[read-failed]token, so readers that already tolerate a leading token are unaffected.pr-readiness-sweep.ymldetects that token on the single status-description read it already makes in its no-evidence branch (zero extra REST reads), and for those PRs runs the new.github/scripts/republish_withheld_verdict.pyinstead of dispatching the aggregator.republish_withheld_verdict.pyaskspr_status.py --disposition-gate(the oneevaluate_reviewer_markerspredicate that both PR Readiness and the local gate answer from) which advisory lanes are owed, locates each lane's run (same-repo by head SHA; fork via its<!-- ai-review-fork-lane run=<id> -->marker, as the human-override handler does) andgh run reruns it. The lane recomputes a fresh verdict and republishes through the same guarded upsert, which already resolves the authority question correctly, so no new trust surface is added.unpublished, readiness drops the token, and the next sweep does not re-fire. A re-run already in flight shows anin_progresscheck-run (scored(not started), no token), so the dispatch→republish window re-fires nothing.MAX_DISPATCH, so a withhold cause that persists (an unreadable head/slot read, a PATCH that keeps failing) would otherwise re-run a paid model review everySTALE_MINUTESforever. Each lane's run now carries itsrun_attempt(already in the runs-list payload for the same-repo path and in theactions/runs/{id}read the fork path already makes), and the sweep stops re-running a lane once its attempt count reachesMAX_RERUN_ATTEMPTS(3), logging a request for a manual re-run instead. Three attempts clears a transient cause while bounding the fail-open cost; it never reaches GitHub's per-run 50-rerun platform limit.Backwards compatibility
Compatible: the change only adds a new recovery path and a new leading status-description token. No field is rejected, renamed or removed; the sweep's existing modes and the readiness scoring are unchanged, and the new token mirrors the pre-existing
[read-failed]token that status-description readers already tolerate.Removes nothing: no path is hidden, no record or key is deleted, renamed or validated more strictly — the diff only adds a token prefix, a sweep branch, and a new script.
Tests
test/test_republish_withheld_verdict.py— unit tests for the new script: owed-lane parsing from the disposition gate, same-repo run lookup by head SHA, fork run lookup via the lane marker, thegh run reruninvocation, the lane→workflow mapping pinned against the override handler's table, fail-closed no-op when the gate returns nothing or an empty set, theMAX_RERUN_ATTEMPTSbound (same-repo and fork paths stop re-running at the cap and still re-run below it), and the same-repohead_repository.full_namebinding that rejects a run in a sibling head repository.test/test_pr_readiness_sweep.py— behavioural: a[verdict-unpublished]pending re-runs the owing lane and does NOT dispatch the aggregator; plus a cross-file pin that the token literal PR Readiness stamps and the token the sweep classifies on stay identical.test/test_readiness_sweep_scan.py— the sweep's sparse-checkout list includes the republisher and the disposition-gate scripts (a dropped entry would be a runtime ImportError only hit in production), and the recovery re-runs the lane via the republisher rather than dispatching the aggregator.Manual verification
N/A — the sweep and readiness bash paths are exercised end-to-end by the test modules above (they run the actual sweep script against fixtured repository states), and the new script is driven with
ghand the disposition gate replaced by stubs. CI's actionlint lane is the authoritative workflow-syntax check.Related Issues
Fixes #13100
Pattern harvest
Not generalizable: this is a specific missing recovery edge in the readiness self-heal engine (a lane that is
successyet owes a verdict), not an instance of a broad defect class — the fix is to add the one recovery branch that was absent, reusing existing predicates and mechanisms.Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)