Skip to content

fix(ci): re-fire a readiness re-check hold once every check has completed - #37841

Merged
KiroCrewCaptain merged 1 commit into
mainfrom
crew/linuxflackyguard/readiness-stuck-pending
Nov 16, 2026
Merged

KiroCrewCaptain merged 1 commit into
mainfrom
crew/linuxflackyguard/readiness-stuck-pending

Conversation

@bolichen97

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A PR Readiness re-check hold no longer stays pending forever once every check on the head has completed.

PR Readiness writes readiness check(s) changed since evaluation; re-checking <lane> while a monitored lane runs, and relies on the self-heal sweep to re-fire it when that lane's completion shows up as check evidence. When the hold runs late (after the lane's check-runs finished, before its workflow run finished), every completion on the head predates the sweep's evidence floor. The sweep then logs not re-firing on every tick. PR #21258 sat on re-checking Sensitive Change Review for 16 days with every check complete.

Why it matters

PR Readiness is the one required status. A frozen pending blocks a merge that is ready, and today only a push or a manual gh workflow run pr-readiness.yml clears it.

Not a goal

What changed (motivation → approach → change)

  • Symptom → cause: the sweep's pending arm re-fires only on check evidence newer than the verdict, or on a [read-failed] stamp. A late hold has neither, so nothing re-fires it.
  • Approach: treat a hold on a settled head like the read-failure class: once it is STALE_MINUTES old, the rollup read was complete and no check-run on the head is still running, re-fire the recompute. The recompute replaces the hold's description, so it fires once; a recompute that writes a hold again is retried at most once per STALE_MINUTES. A hold with a check-run still running stays on the evidence path.
  • Change: pr-readiness-sweep.yml adds readiness_is_recheck_hold (prefix match on the description, read only on the existing no-evidence branch) and the new re-fire reason. docs/ci/ci-and-reviews.md describes the rule.

Backwards compatibility

Compatible: the sweep only gains one more re-fire case; every case that re-fired before still does.

Removes nothing: no key, path, status or check is removed or tightened.

Tests

test/test_pr_readiness_sweep_recheck_hold.py runs the real sweep block against a gh stub:

  • a hold with every check done, 186 s past the newest completion (the PR #21258 shape), is re-fired once;
  • a hold with a check-run still running is not (a partial rollup read does not qualify either);
  • a hold younger than STALE_MINUTES is not, and costs no status read;
  • an ordinary lane-pending on a settled head still is not;
  • the next tick after the recompute publishes a real verdict dispatches nothing;
  • every re-checking description in pr-readiness.yml starts with the sweep's prefix.

Proof: PROVEN test/test_pr_readiness_sweep_recheck_hold.py::test_a_hold_on_a_head_with_every_check_completed_is_refired, test/test_pr_readiness_sweep_recheck_hold.py::test_the_recompute_that_replaces_the_hold_ends_the_retry, test/test_pr_readiness_sweep_recheck_hold.py::test_every_hold_the_publisher_writes_carries_the_sweep_prefix (head 3cc2eea)

Determinism: 20/20 at -n0, 10/10 at -n 4 beside test_pr_readiness_sweep.py and test_pr_readiness_sweep_debounce.py, and a shuffled order. All 120 sweep and scanner tests pass.

Manual verification

On 2026-11-16 05:49 UTC the open heads in this state were re-dispatched with the documented command (no check skipped): PR #21258, PR #37806, PR #37298, PR #37789, PR #37616, PR #37787. Each got a real verdict within 30 s, which is what this sweep rule will now do on its own for the frozen case.

Related Issues

Part of #37840

Sibling sweep

Swept: git grep -n "re-checking" -- .github/workflows/pr-readiness.yml
Sibling: .github/workflows/pr-readiness.yml:812 -- not affected -- the hold step writes the prefix the sweep now matches
Sibling: .github/workflows/pr-readiness.yml:2384 -- not affected -- the publish re-check writes the same prefix, pinned by the new test

Pattern harvest

Not generalizable: the no-evidence pending classes are an explicit list in one sweep arm, and the new test pins every writer of this one.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@bolichen97
bolichen97 requested review from a team and KiroCrewCaptain as code owners November 16, 2026 06:08
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 16, 2026
@github-actions

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 3cc2eea9ad29090e8daedfc5d2c01f997012a643 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 3cc2eea9ad29090e8daedfc5d2c01f997012a643 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: a CI-only fix for a pending status that never clears. The new retry fires only once every check is complete, the full rollup was read and the hold is STALE_MINUTES old, and it repeats at most once per STALE_MINUTES. It is reverted cleanly.

Notes

  • A hold whose lane's workflow run never closes, with no check-run left running, is retried every STALE_MINUTES for as long as that run stays open. The doc says so; keep an eye on dispatch volume if runs hang for a long time.

[DESIGN-REVIEWED] 3cc2eea

Description read

[DESCRIPTION-READ] e98c3b538042091450b54f07e8dfde4d45951ae3f91ff9331d80bd05fe513588

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of 3cc2eea9ad29090e8daedfc5d2c01f997012a643 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Nothing blocks. It is a fix with a named defect (PR #21258 stuck for 16 days), it covers every writer of the hold from one place in the sweep, and the new branch runs only when no other evidence will ever arrive.

What this change ships

Inventory (4 items) — 4 justified

Intent: let a PR Readiness re-check hold clear by itself once every check on the head has finished, instead of needing a push or a manual dispatch. FIX.

  1. The sweep re-fires a re-checking <lane> pending once it is STALE_MINUTES old, the rollup read was complete and no check-run is still running — justified
  2. A hold with a check-run still running, or read from a partial rollup, keeps waiting on the evidence path as before — justified
  3. The CI review doc gains a paragraph describing the rule and the PR feat: add AWS Fargate session environments #21258 case — justified
  4. A test checks that every re-checking description in pr-readiness.yml starts with the prefix the sweep matches — justified

[FIRST-PRINCIPLES-REVIEWED] 3cc2eea

Description read

[DESCRIPTION-READ] 2daf12ec446bc762cbbedeec01846f53fee57c643d3609525b7d5a4af9689abd

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 3cc2eea9ad29090e8daedfc5d2c01f997012a643 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 3cc2eea

Verdict parsed from the review's SHA-scoped output markers for commit 3cc2eea9ad29090e8daedfc5d2c01f997012a643.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 3cc2eea9ad29090e8daedfc5d2c01f997012a643 false-positive: <reason, 20 to 500 characters>

@github-actions

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 3cc2eea9ad29090e8daedfc5d2c01f997012a643 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 3cc2eea

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 3cc2eea9ad29090e8daedfc5d2c01f997012a643 false-positive: <reason, 20 to 500 characters>

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Nov 16, 2026
@bolichen97
bolichen97 enabled auto-merge (squash) November 16, 2026 07:18

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I reviewed head 3cc2eea and walked the new sweep branch. It fires only on a re-check hold at least STALE_MINUTES old, with the full rollup read and no check-run still running. Every description pr-readiness.yml writes for a hold starts with the prefix the sweep matches, and the read-failure and withheld-verdict paths behave as before. The new tests and the existing sweep, debounce and scanner tests pass on this head. The core re-fire tests fail against main's sweep, and the running-check test fails when its guard is removed, so they do test the change. A recompute on a settled head writes its own verdict description, so the retry is bounded to once per STALE_MINUTES while a lane's run is still open, and I found nothing that blocks.

@KiroCrewCaptain
KiroCrewCaptain merged commit 78db723 into main Nov 16, 2026
96 checks passed
@KiroCrewCaptain
KiroCrewCaptain deleted the crew/linuxflackyguard/readiness-stuck-pending branch November 16, 2026 07:21

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the full diff.

Sensitive change review (pr-readiness-sweep.yml, the sweep behind the one required status)

  • What changed: one more re-fire case. A PR Readiness pending whose description starts readiness check(s) changed since evaluation; is re-fired once it is STALE_MINUTES old, but only with zero check-runs running and a complete rollup read.
  • Before -> after: such a hold stayed pending until a push (#21258: 16 days) -> the sweep dispatches a recompute.
  • Worst case if wrong: extra recompute dispatches, at most one per STALE_MINUTES per PR while a hold persists. A recompute re-derives the verdict from live checks, so it cannot publish a green by itself.
  • Evidence checked: the quoted case prefix matches literally and fails closed on an empty description; the if/elif/else keeps the read-failure branch and the not re-firing default exactly as before; a non-integer $running makes the test false and falls through. The new test pins the prefix against every hold site in pr-readiness.yml, plus the running, young, ordinary-pending and replaced cases.
  • No regression: every case that re-fired before still does; nothing is tightened.
  • Undo: revert the squash commit.

PR Readiness green on this head.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants