Skip to content

ci: release the matrix when a Fast Gate run's status lags its finished green jobs - #25609

Merged
iamwhatever merged 1 commit into
mainfrom
fix/await-fast-gate-job-rollup-24890
Oct 29, 2026
Merged

iamwhatever merged 1 commit into
mainfrom
fix/await-fast-gate-job-rollup-24890

Conversation

@CrysisDeu

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Await Fast Gate stops failing a PR whose Fast Gate jobs all passed but whose run-level status has not caught up.

The barrier reads only the Fast Gate run's status. On fork PR #9750 all 16 Fast Gate jobs finished success by 13:32:33, but the run kept reading in_progress until about 14:05. The barrier hit its 720s budget at 13:43:15 and failed closed, so the whole test matrix was skipped and Coverage Gate went red.

Why it matters

A green gate fails the matrix, and on a fork PR the author cannot rerun the job: the only recovery is a new push or a maintainer rerun.

Not a goal

  • Changing the 180s / 720s budgets, the supersession proof, or the action_required handling.
  • Polling jobs on every tick: the jobs are read once, only when the budget is already spent.

Intended? / Repro

Intended? No. The barrier's job is to confirm the gate passed (.github/scripts/await-fast-gate.sh:1-5 on main); a run whose every job passed has passed, and GitHub's rollup lag is not a verdict on the code.

Repro: Await Fast Gate job 122218407972 (run 40511382311) logs Fast Gate still 'in_progress' after 721s, while Fast Gate run 40511382148 shows every job success by 13:32:33 and the run updated_at 14:05:41.

What changed (motivation → approach → change)

Symptom: a fail-closed exit on a green gate. Cause: run-level status only. Change: at the TOTAL budget, before failing closed, check the run's jobs once.

  • .github/scripts/await-fast-gate.sh: new jobs_all_passed (a GET of runs/{id}/attempts/{attempt}/jobs). It answers yes only when the attempt has jobs, all of them are listed, every one is completed with success or skipped, at least one is success, and the newest finished at least ROLLUP_SETTLE (60s) ago, so a job that needs: the others has had time to appear. Then the barrier exits 0 with a warning naming the lag. A failed read, an unfinished or failed job, all-skipped, or a fresh finish all keep the existing fail-closed exit. A run awaiting fork approval is never checked.
  • docs/ci/ci-and-reviews.md: the await-fast-gate row states the rule.

Security

The barrier stays read-only (actions: read, GET only). It now trusts GitHub's own per-job conclusions, the same data the run-level conclusion is rolled up from; nothing a PR author writes is read.

Backwards compatibility

Compatible: every path that passed or failed before does the same, except a run stuck in_progress at 720s whose jobs all passed, which now releases the matrix.

Removes nothing.

Tests

test/test_await_fast_gate_poll.py TestARunWhoseStatusLagsItsJobs, through the real script with the stubbed gh and fake clock:

  • all jobs green and settled → exit 0, the lag warning, only GET calls, and the jobs read targets runs/16/attempts/1/jobs;
  • a running job, a failed job, or only skipped jobs → still fails closed;
  • no jobs → fails closed;
  • jobs finished seconds ago → not trusted yet, fails closed;
  • an unreadable jobs list → fails closed and says so.

The existing test_a_run_that_stays_in_progress_fails_at_the_total_budget still passes: its jobs read fails, so it fails closed as before. On main the first new test fails (exit 1).

Proof: not provable -- other: this host runs no local tests; CI runs the new tests

Manual verification

bash -n on the script. The jq verdict expression was run by hand against job lists for each case above (green settled → yes 2; failed, running, empty, just finished → no). Static gates: black, flake8, scripts/docs_lint.py, scripts/check_comment_history.py.

Related Issues

Fixes #24890

Sibling sweep

Swept: git grep -n "await-fast-gate.sh" -- .github
Sibling: .github/workflows/ci.yml await-fast-gate job -- not affected (runs this file unchanged)
Sibling: .github/scripts/cancel-superseded-run.sh -- not affected (runs this file as a probe; a release exits 0, which it already treats as nothing to cancel)

Pattern harvest

Not generalizable: one barrier.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributed under the project's Apache-2.0 license.

@CrysisDeu
CrysisDeu requested review from a team and KiroCrewCaptain as code owners October 29, 2026 03:14
@CrysisDeu
CrysisDeu requested a review from mbriones98 October 29, 2026 03:14
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 29, 2026
@github-actions

github-actions Bot commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ⏭️ skipped

Revision 1c57345792804d4ee2c4bfdd9b7903e2b978554b ships no reviewable capability (docs, tests or generated files only), so there is nothing to inventory. Does not block merge.

@github-actions

github-actions Bot commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 1c57345792804d4ee2c4bfdd9b7903e2b978554b touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 1c57345792804d4ee2c4bfdd9b7903e2b978554b — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks. Releasing the matrix does not replace Fast Gate's own check, which still has to conclude green. A wrong release only costs a wasted matrix run; it does not admit unjudged code.

Notes

  • The 60s ROLLUP_SETTLE assumes GitHub creates a job that needs: the others within 60s, but the same GitHub lag that left the run status stale could also delay that. If Fast Gate has an aggregator job that can fail, that window is where a false release would come from. It is worth saying so in the doc row.

[DESIGN-REVIEWED] 1c57345

Description read

[DESCRIPTION-READ] e94e3b6734505d408a0e87c60204619ae3242c55b2de995d5fd34adfd6fa951b

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 1c57345792804d4ee2c4bfdd9b7903e2b978554b — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 1c57345

Verdict parsed from the review's SHA-scoped output markers for commit 1c57345792804d4ee2c4bfdd9b7903e2b978554b.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 1c57345792804d4ee2c4bfdd9b7903e2b978554b false-positive: <reason, 20 to 500 characters>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 29, 2026
@github-actions

github-actions Bot commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 1c57345792804d4ee2c4bfdd9b7903e2b978554b and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 1c57345

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 1c57345792804d4ee2c4bfdd9b7903e2b978554b false-positive: <reason, 20 to 500 characters>

@CrysisDeu
CrysisDeu force-pushed the fix/await-fast-gate-job-rollup-24890 branch from 402ee9f to 8cbeb99 Compare October 29, 2026 03:34
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 29, 2026
@CrysisDeu
CrysisDeu force-pushed the fix/await-fast-gate-job-rollup-24890 branch from 8cbeb99 to a8d4bbb Compare October 29, 2026 08:27
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 29, 2026
@CrysisDeu
CrysisDeu force-pushed the fix/await-fast-gate-job-rollup-24890 branch from a8d4bbb to 6f18e8b Compare October 29, 2026 11:26
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 29, 2026
@CrysisDeu

CrysisDeu commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor Author

GPT 6.1 BLOCK at 6f18e8b (a green partial rerun releases the matrix over a still-failing job from attempt 1): fixed in 35a74a1, using the fix the finding names.

  • .github/scripts/await-fast-gate.sh jobs_all_passed now trusts a job list only on attempt 1. On a later attempt it warns ("on attempt N; a rerun's job list leaves out earlier attempts") and returns 1, so the barrier fails closed as it did before this PR.
  • test_a_rerun_attempt_still_fails_closed: a run on run_attempt: 2 whose listed jobs are all green still exits 1 and never reads /jobs.
  • The await-fast-gate row in docs/ci/ci-and-reviews.md now says the jobs check runs on a first attempt only.

@CrysisDeu
CrysisDeu force-pushed the fix/await-fast-gate-job-rollup-24890 branch from 6f18e8b to 35a74a1 Compare October 29, 2026 12:25
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 29, 2026
@CrysisDeu
CrysisDeu force-pushed the fix/await-fast-gate-job-rollup-24890 branch from 35a74a1 to 1c57345 Compare October 29, 2026 13:18
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 29, 2026

@iamwhatever iamwhatever left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve at 1c57345792804d4ee2c4bfdd9b7903e2b978554b.

The new release path runs only when the 720s budget is spent and the run is neither completed nor awaiting approval. It reads the jobs of attempt 1 only. It requires the listed count to match total_count, every job to be success or skipped with at least one success, and the newest job to have finished at least 60s earlier. Any read or parse failure keeps the old fail-closed exit (await-fast-gate.sh#L184-L233, #L332-L345). The earlier GPT BLOCK about partial reruns is fixed by the attempt-1 guard. Every call is still a GET, and the PR changes no permissions. CI and every AI lane are green on this head.

AI findings judged non-blocking

  • Design Review (Note): the 60s settle may not cover a late-created needs: aggregator. Does not block: Fast Gate forbids needs: on its gate jobs (fast-gate.yml#L77). A false release would also only start the matrix, because Fast Gate's own conclusion still gates readiness.

— crew-pr-reviewer (r135)

@iamwhatever
iamwhatever enabled auto-merge (squash) October 29, 2026 16:54
@iamwhatever
iamwhatever merged commit bf0d414 into main Oct 29, 2026
94 checks passed
@iamwhatever
iamwhatever deleted the fix/await-fast-gate-job-rollup-24890 branch October 29, 2026 16:56
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ci: Await Fast Gate fails closed when the Fast Gate run status lags its finished jobs

2 participants