Skip to content

fix(readiness): hold a fork PR into a non-default base for a maintainer - #25224

Closed
mbriones98 wants to merge 2 commits into
mainfrom
fix/fork-release-hold-18530
Closed

mbriones98 wants to merge 2 commits into
mainfrom
fix/fork-release-hold-18530

Conversation

@mbriones98

@mbriones98 mbriones98 commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A fork PR into a release branch gets a final readiness verdict that names why it is held, instead of a pending verdict the sweep re-checks forever.

A fork PR whose base is a release branch kept PR Readiness at pending for the life of its head, and the self-heal sweep re-dispatched a recompute every tick. #18441 got 17 identical publications in six and a half hours: [read-failed] 8 readiness check(s) still pending; waiting on Internal Content Scan (not started) (+7 more).

Why it matters

An outside contributor backporting a fix to a release branch had no way to reach a final verdict, and each affected PR cost two wasted readiness runs per half hour. The only workaround so far was re-opening the PR from a same-repo branch, which needs push access.

Not a goal

  • Running the fork scan and AI reviews on fork PRs into release branches (widening fast-gate.yml). That is a separate decision; this PR changes no fork permission or token scope.
  • Letting these PRs pass on their own. They stay held for a maintainer.

What changed (motivation → approach → change)

This fixes the loop but keeps these PRs held for a maintainer.

Symptom → cause. fast-gate.yml only runs on PRs to main, and all seven Stage-2 fork lanes fire on Fast Gate completing, so on a fork PR into another base none of them ever runs. pr-readiness.yml still scored them (not started). The supersession gate then found no lane comment at all, and readiness stamped that as a read failure. The sweep re-fires any [read-failed] pending on age, and the recompute re-derived the same pending.

Change. A fork PR into a non-default base now gets a final failure: fork PR into a non-default base needs maintainer review. It never reaches passed: none of the fork scan or review coverage runs on it while its base is another branch. If GitHub retargets it to main, the recompute on that event reads it as an ordinary fork PR. The sweep re-fires a failure only when a check, review or comment lands after the verdict, never on age, so the loop stops.

Situation on a fork PR into a non-default base Before After
Fork lane whose Fast Gate trigger never ran pending (not started) skipped (only runs on PRs to main)
Verdict a lane posted while the base was main (red included) read still read and still counts
Supersession gate cannot evaluate (no lane, unreadable, timed out) [read-failed] pending still listed; the hold publishes failure first, so no [read-failed] token
Overall verdict pending forever failure, named hold

Fork PRs into main and same-repo PRs into any base are unchanged.

Backwards compatibility

Compatible: only fork PRs into a non-default base change, and they could never reach a verdict before.

Removes nothing: no lane, key or check is removed; lanes that cannot run on this base are listed as skipped, and the PR is held rather than passed.

Tests

test/test_pr_readiness_fork_base.py covers the hold being final, named and free of [read-failed]; the PR not passing even with every lane green; a red lane and a lost reviewer block from before a base change still counting; a real supersession read failure still showing; the sweep not re-firing the hold, while it did re-fire the old pending; and fork PRs into main and same-repo release PRs being unchanged, including their [read-failed] pending when supersession cannot be evaluated. The sweep tests skip on Windows and without GNU date, as the sweep suite does.

A differential run of the evaluate step over 144 PR shapes (fork or same-repo, base, supersession result, dropped block, Fast Gate ran or not, lane results) matched main on every shape that is not held.

Proof: PROVEN test/test_pr_readiness_fork_base.py::TestAForkPullRequestIntoANonDefaultBaseIsHeld::test_the_hold_is_terminal_named_and_not_a_read_failure, test/test_pr_readiness_fork_base.py::TestAForkPullRequestIntoANonDefaultBaseIsHeld::test_it_does_not_pass_even_with_every_lane_green, test/test_pr_readiness_fork_base.py::TestAForkPullRequestIntoANonDefaultBaseIsHeld::test_a_red_lane_posted_before_the_base_changed_is_kept, test/test_pr_readiness_fork_base.py::TestAForkPullRequestIntoANonDefaultBaseIsHeld::test_a_dropped_block_is_kept, test/test_pr_readiness_fork_base.py::TestAForkPullRequestIntoANonDefaultBaseIsHeld::test_a_real_supersession_read_failure_still_surfaces (head 0ebde6a)

The readiness, sweep, supersession and fork-workflow suites pass (875 tests); actionlint, ruff, black and the comment-history gate are clean.

Manual verification

N/A — the evaluate step is executed end to end by the test harness with stubbed GitHub reads; nothing was dispatched on a live PR.

Known follow-ups

Related Issues

Fixes #18530
Replaces #24936

Sibling sweep

Swept: git grep -n "fast-gate.yml" -- .github/workflows/
Siblings: none -- the fork lane specs are read in one place, the evaluate step of pr-readiness.yml, changed here

Pattern harvest

Rule candidate: review-prompt
Pattern: an "unknown" answer from a gate stamped as a read failure on a population that structurally can never exist, so an age-based retry loops forever

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Fast Gate only runs on PRs to the default branch, and every Stage-2 fork
lane fires on Fast Gate completing, so a fork PR into a release branch gets
no run of any of them. The evaluate step scored those lanes (not started)
and read the supersession gate's empty answer as a read failure, so the
verdict stayed at a [read-failed] pending the sweep re-fired a recompute
for on every tick.

Such a PR now gets a terminal failure, 'fork PR into a non-default base
needs maintainer review', with no [read-failed] token, so the sweep no
longer re-fires it on age. It never reaches passed: none of the fork scan
or review coverage runs on it while its base is another branch. A fork lane
whose trigger never ran for the head is listed as skipped; a verdict a lane
already posted while the base was the default branch is still read, a red
one included. The hold is a blocker, and a blocker publishes failure before
the pending branch can add the read-failure token, so a supersession read
failure on such a PR is still listed but cannot loop the sweep. Every other
PR shape is evaluated as before.

Fixes #18530
@mbriones98
mbriones98 requested review from a team and KiroCrewCaptain as code owners October 28, 2026 11:56
@mbriones98
mbriones98 enabled auto-merge (squash) October 28, 2026 11:56
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 28, 2026
@github-actions

github-actions Bot commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ✅ PASS

UX-level review of 66d3d24b54290e6f0055c21bd35bf43e39a5af69 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

UX-Verdict: PASS

Nothing to check. The diff changes only a CI workflow and tests, with no dashboard control, string or state.

[UX-REVIEWED] 66d3d24

Description read

[DESCRIPTION-READ] 4602ce24da027a6dc9a8e9120a3c0956346d8bf58d4d781a494d4de4062e6d52

That sha256 names everything this verdict read: the PR description, plus the 43 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of 66d3d24b54290e6f0055c21bd35bf43e39a5af69 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Nothing blocks. The hold reuses the existing stacked carve-out, sits at mechanism level, and the cause it leaves alone (widening fast-gate.yml) is excluded under ## Not a goal.

What this change ships

Inventory (5 items) — 4 justified

Intent: let a fork PR into a release branch reach a final, named readiness verdict instead of staying pending forever. This is a FIX (#18530).

  1. On a fork PR into a non-default base, a fork lane whose Fast Gate trigger never ran now shows as skipped "(only runs on PRs to main)" instead of pending "(not started)" — justified
  2. The overall verdict for such a PR is now a final failure that holds it for a maintainer, so the sweep no longer re-fires it on age — justified
  3. The status text for the hold names it and counts the other blockers and approval waits — justified
  4. Before posting, the publish step re-reads the PR's base and drops any verdict that was scored for a base the PR no longer has, on every PR — undeclared (in the commit message, not the description)
  5. The publish step gets a new BASE_REF env input, which only item 4 reads — justified

[FIRST-PRINCIPLES-REVIEWED] 66d3d24

Description read

[DESCRIPTION-READ] 63822788c551f4f22abc8be926c249a322f68fa6bbdcfc8144bac26dc8d0de1f

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 66d3d24b54290e6f0055c21bd35bf43e39a5af69 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: a final, named failure ends the age-based re-fire loop, and the PR still cannot pass without a maintainer.

Notes

  • The diff also adds a stale-base guard to the publish step (current_base != BASE_REF → exit 0), and the description never mentions it. Add one line on why it is needed: the hold must not land on a PR that was retargeted to main mid-run, or a run for the old base could publish over the new one.
  • "Held for a maintainer" names no way to lift the hold. A maintainer still has to bypass branch protection or re-open the PR from a same-repo branch, as before. Say so in the follow-ups so the hold doesn't read as reviewable in-product.

[DESIGN-REVIEWED] 66d3d24

Description read

[DESCRIPTION-READ] ddc3e5eb0e53064b8b6046e56fdefad8ddcd781c0dcb25059f9cad928dda4e24

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 66d3d24b54290e6f0055c21bd35bf43e39a5af69 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 66d3d24

Verdict parsed from the review's SHA-scoped output markers for commit 66d3d24b54290e6f0055c21bd35bf43e39a5af69.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 66d3d24b54290e6f0055c21bd35bf43e39a5af69 false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — 🔴 changes requested (blocking)

GPT 6.1 found at least one blocking issue that must be resolved before merging 66d3d24b54290e6f0055c21bd35bf43e39a5af69. Adjudication upheld 1 of 1 blocking finding(s); those must be resolved before merging.

This comment is updated in place on each push.

BLOCKING -- .github/workflows/pr-readiness.yml:2424 -- Failed base re-check preserves a stale green verdict

current_base="$(gh_retry gh pr view "$PR" --repo "$REPO" --json baseRefName --jq '.baseRefName')"
Trigger: Retarget an already-green fork PR from main to a release branch; its edited run evaluates the maintainer hold, but this final API read fails after retries.
Outcome: set -e exits before the status POST, leaving the unchanged head’s previous successful PR Readiness status and passed label instead of enforcing the hold. No failure handler replaces that status.
Anchor: defect
Fix: Catch the read failure and publish a non-success [read-failed] pending status; skip publication only when a successful read establishes that the base changed.

[BLOCK-MERGE] 66d3d24
[GPT-REVIEWED] 66d3d24

Adjudication (Opus 5.5) — is blocking on each finding proportionate?

F1 — HIGH, silently. A fork PR can skip the maintainer hold: the green PR Readiness status stays on the SHA even though the PR's base no longer matches the one that verdict was scored for. Conditions confirmed:

  • The step runs under set -euo pipefail (pr-readiness.yml:2110).
  • current_base="$(gh_retry …)" at :2424 has no failure branch, so a retry-exhausted read exits the step before the POST at :2436.
  • An edited run writes no earlier pending status; the hold step only runs on workflow_run/in_progress (:645-649). Recovery is only a manual re-run.

Real fix cost: one if/else that writes a [read-failed] pending status. The same step already does exactly that at :2386-2388, so it needs no new state or mechanism.

[ADJUDICATION] 66d3d24b54290e6f0055c21bd35bf43e39a5af69 total=1 uphold=1 downgrade=0
UPHOLD F1 .github/workflows/pr-readiness.yml:2424 reason=harm-warrants-remedy
[GPT-ADJUDICATED] 66d3d24b54290e6f0055c21bd35bf43e39a5af69

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 66d3d24b54290e6f0055c21bd35bf43e39a5af69 false-positive: <reason, 20 to 500 characters>

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 28, 2026

@iamwhatever iamwhatever left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes for one missing follow-up issue; the code itself looks sound.

Reviewed head 0ebde6aecc73dea4c1679bd0853d85c73659e3b6.

The fix is right: on a fork PR into a non-default base, a Fast Gate-keyed lane that never ran is listed as skipped, and the maintainer hold goes into failed[] after the supersession gate, so the verdict is a terminal failure the sweep does not re-fire on age. Fork-into-main and same-repo shapes are untouched, and the new tests cover the hold, retained red lanes and dropped blocks, and the sweep.

Blocker: the second Known follow-up has no linked issue. A fork PR retargeted from a release branch to main gets no new Fast Gate run, because fast-gate.yml runs on the default pull_request types and a retarget is edited. Its Stage-2 lanes then sit at (not started) pending until the author pushes, which is close to the loop this PR fixes. The new code comment says the hold lifts on retarget, which is true, but the PR then waits on lanes that will not start. Please either handle it here or open one issue for it and link it in the description.

The first Known follow-up (transient read failure before the hold) self-heals on the next sweep and needs no issue.

AI findings judged non-blocking:

  • Design Review (note): the hold cannot be cleared in-product, so a maintainer must bypass PR Readiness; suggests saying so in the description. Wording only, and no worse than the old endless pending (hold).

All current-head checks are green and every AI lane passed or skipped.

— crew-pr-reviewer (r133)

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: passed Eligible automated validation passed for the current revision labels Oct 28, 2026
The publish step re-checked only the head before writing the verdict.
A base change keeps the SHA, so a run that evaluated a PR while its
base was main could finish after the base changed and publish success
over the maintainer hold a newer run had already written for a fork PR
into another branch.

The step now reads the PR's base after every other read in the step,
right before the status POST, and publishes nothing when it differs
from the base the verdict was scored for. The head is still checked at
the top of the step. The run for the base change publishes instead.
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 30, 2026
@bolichen97

Copy link
Copy Markdown
Contributor

[operator: bolichen97] Continuing this in #31446; thanks for the start.

bolichen97 added a commit that referenced this pull request Nov 6, 2026
Fast Gate only runs on PRs to the default branch, and every Stage-2 fork
lane fires on Fast Gate completing, so a fork PR into a release branch
gets no run of any of them. The evaluate step scored those lanes (not
started) and read the supersession gate's empty answer as a read
failure, so the verdict stayed at a [read-failed] pending the sweep
re-fired a recompute for on every tick.

Such a PR now gets a terminal failure, 'fork PR into a non-default base
needs maintainer review', with no [read-failed] token, so the sweep no
longer re-fires it on age. A fork lane whose trigger never ran for the
head is listed as skipped; a verdict a lane already posted while the
base was the default branch is still read, a red one included.

The verdict is scored for one base as well as one head, and a base
change keeps the SHA. The publish step reads the base right before the
status POST and publishes nothing when it moved; when that read fails,
a success is published as a stamped pending instead of standing for a
base it was not scored for. Every other PR shape is evaluated as before.

Builds on #25224.

Fixes #18530

Co-authored-by: Matthew Briones <50031759+mbriones98@users.noreply.github.com>
bolichen97 added a commit that referenced this pull request Nov 6, 2026
…er (#31446)

Fast Gate only runs on PRs to the default branch, and every Stage-2 fork
lane fires on Fast Gate completing, so a fork PR into a release branch
gets no run of any of them. The evaluate step scored those lanes (not
started) and read the supersession gate's empty answer as a read
failure, so the verdict stayed at a [read-failed] pending the sweep
re-fired a recompute for on every tick.

Such a PR now gets a terminal failure, 'fork PR into a non-default base
needs maintainer review', with no [read-failed] token, so the sweep no
longer re-fires it on age. A fork lane whose trigger never ran for the
head is listed as skipped; a verdict a lane already posted while the
base was the default branch is still read, a red one included.

The verdict is scored for one base as well as one head, and a base
change keeps the SHA. The publish step reads the base right before the
status POST and publishes nothing when it moved; when that read fails,
a success is published as a stamped pending instead of standing for a
base it was not scored for. Every other PR shape is evaluated as before.

Builds on #25224.

Fixes #18530

Co-authored-by: Matthew Briones <50031759+mbriones98@users.noreply.github.com>
@github-actions github-actions Bot added the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Nov 6, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor

Thanks for this fix. Its whole change is now on main through #31446 (merged 2026-11-06, commit 24f8a91), which says it builds on this PR, and #18530 is closed.

On main, .github/workflows/pr-readiness.yml holds a fork PR into a non-default base for a maintainer, lists its Fast Gate-keyed lanes as skipped (only runs on PRs to $DEFAULT_BRANCH), and ignores a publish scored for a base the PR no longer has ("Ignoring publish for stale base"). test/test_pr_readiness_fork_base.py and the base fixtures in test/test_pr_readiness_publish.py are there too.

So I'm closing this one as superseded. Please reopen if I missed a part that did not land.

@iamwhatever iamwhatever closed this Nov 7, 2026
auto-merge was automatically disabled November 7, 2026 10:36

Pull request was closed

@github-actions github-actions Bot removed the readiness: action required A blocking check or review needs attention label Nov 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

merge conflict Branch has merge conflicts with its base — author must resolve before merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ci(readiness): a fork PR into a release branch never resolves, and the sweep re-fires a recompute forever

3 participants