Skip to content

fix(ci): read an empty head_sha run listing through check-runs - #28431

Closed
dodgeblaster wants to merge 1 commit into
mainfrom
fix/23245-empty-run-listing
Closed

dodgeblaster wants to merge 1 commit into
mainfrom
fix/23245-empty-run-listing

Conversation

@dodgeblaster

@dodgeblaster dodgeblaster commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: CI stops treating an empty actions/runs?head_sha= answer as proof that a head has no runs, so a green lane is not read as "(not started)" and a lane run is not reported as missing when it exists.

The head_sha-filtered runs listing is a search that has answered total_count: 0 for a head whose own check-runs named 30 runs (#23245). Every reader in .github/** took that empty answer at face value:

  • PR Readiness read every lane as (not started) and held the verdict at pending; its success re-check found no open lane and could publish success over a lane that had just restarted.
  • ai-review-human-override.yml reported "Fast Gate has not run" on a fork PR (so the override was not bound to the fork lane run), and "no workflow run found" on a same-repo PR.
  • republish_withheld_verdict.py reported the lane run "could not be located" and asked for a manual re-run.

The pr_status.py reader was fixed in #23253; this PR covers the readers that live in .github/**.

Why it matters

The required PR Readiness status can sit at pending on a green PR until a later recompute happens to get a non-empty answer. An override or republish that cannot find its run leaves a lane blocked or unpublished and asks a person to re-run it by hand. Each of these happens on a head whose runs exist.

Not a goal

  • Changing why GitHub's filtered listing comes back empty, or reading runs from a different primary source. The listing is still the first read.
  • await-fast-gate.sh, which polls its listing every 10s and re-reads an intermittent empty answer.
  • Display-only readers (fix_loop_metrics.py, Issue Radar list_pr_workflow_runs), which decide nothing from an empty answer.

What changed (motivation → approach → change)

Symptom: an empty listing reads as "no runs". Root cause: each reader has only one source for a head's runs. Change: when the listing is empty, read the head's check-runs. An Actions job's check-run links its run in details_url (.../actions/runs/<run id>/job/<job id>). Each distinct run is fetched and kept only when it ran for that event on that exact head. This is the same idea as #23253: the head's own check data outranks an empty listing.

  • pr-readiness.yml: a new gh_head_runs <repo> <sha> <event> in the shared gh-retry.sh helper. The evaluation's pull_request and dynamic reads and the publish step's two success re-check reads all go through it. If a named run cannot be fetched, the evaluation reads could not be evaluated, never (not started).
  • ai-review-human-override.yml: a head_runs_from_check_runs fallback behind the fork Fast Gate binding and the same-repo lane-run lookup. If a named run cannot be read, the result is a read-failure fail_lane, never "has not run" or "not found".
  • republish_withheld_verdict.py: _same_repo_run_id and _fork_trigger share a new _head_runs. It returns None (unknown) when a named run cannot be fetched.
  • docs/ci/ci-and-reviews.md now documents the rule.

A non-empty listing is used exactly as before and makes no extra request. An empty listing costs one check-runs read, plus one read per run it names that the caller does not already hold: the dynamic (CodeQL) reads skip the runs already on the pull_request page, so a head with no dynamic run costs one check-runs read.

Backwards compatibility

Compatible: a non-empty listing takes the same path as on main, and an empty listing with no Actions job check-runs still reads as "no runs".

Removes nothing: the change only adds a second source behind an empty answer. No path, field, validator or record is taken away.

Tests

  • test_pr_readiness_empty_run_listing.py::TestAnEmptyRunsListingIsNotNoRuns: runs named by the check-runs answer for an empty listing (verdict success). A named run that cannot be read gives could not be evaluated. A named run for another head is not borrowed. A sibling's newer same-named row on a shared head does not hide this PR's runs. A sibling's newer same-named row on a shared head does not hide this PR's runs. A sibling's newer same-named row on a shared head does not hide this PR's runs. An empty listing with no job check-runs is still (not started). A non-empty listing reads no check-runs and fetches no single run.
  • test_pr_readiness_publish.py: a lane restarted behind an empty listing still holds the success. An unreadable named run does not publish success.
  • test_ai_review_workflows_human_override.py: fork and same-repo, an empty listing binds or re-runs the run the check-runs name. An unreadable named run is a read failure. A run for another head does not bind. No job check-runs still reads "has not run" / "not found". The .details_url guard is scoped to the fork lane resolver it protects, because that resolver still reads the lane run only from the marker.
  • test_republish_withheld_verdict.py: fork and same-repo lookups through the check-runs, the four no-readable-run cases, no check-runs read on a non-empty listing, run-id parsing from Actions job links only, and the check-runs read asking for every row (filter=all) rather than the latest per name.

Proof: PROVEN test/test_ai_review_workflows_human_override.py::TestOverrideHandlerReRunsTheBoundForkLaneRun::test_an_empty_listing_binds_the_fast_gate_run_the_check_runs_name, test/test_ai_review_workflows_human_override.py::TestOverrideHandlerReRunsTheBoundForkLaneRun::test_an_empty_listing_reads_every_check_run_not_only_the_latest, test/test_ai_review_workflows_human_override.py::TestOverrideHandlerReRunsTheBoundForkLaneRun::test_a_named_fast_gate_run_that_cannot_be_read_is_a_read_failure, test/test_ai_review_workflows_human_override.py::TestOverrideHandlerReRunsTheBoundForkLaneRun::test_a_same_repo_empty_listing_re_runs_the_lane_run_the_check_runs_name, test/test_ai_review_workflows_human_override.py::TestOverrideHandlerReRunsTheBoundForkLaneRun::test_a_same_repo_empty_listing_re_runs_the_newest_named_lane_run (head 313b92f)

Manual verification

N/A — every changed reader is a workflow step or script that the tests execute against a stubbed gh. I could not reproduce the intermittent empty listing on demand against the live API. Local gate: the related backend tests passed (961). The related frontend run could not start because this worktree has no website/node_modules, and this diff touches no frontend code.

Screenshots / video

N/A — CI workflows and scripts only; no user-visible surface.

Related Issues

Fixes #23245

Sibling sweep

Swept: git grep -nE "head_sha=" -- .github/ scripts/ src/
Sibling: .github/workflows/pr-readiness.yml -- changed here
Sibling: .github/workflows/ai-review-human-override.yml -- changed here
Sibling: .github/scripts/republish_withheld_verdict.py -- changed here
Sibling: src/kiro_crew/builtin_skills/kirocrew-dev/kirocrew-prepare-pr/scripts/pr_status.py -- not affected -- fixed by the rollup-row check in #23253
Sibling: .github/scripts/await-fast-gate.sh -- not affected -- it polls every 10s and re-reads an intermittent empty answer
Sibling: .github/scripts/fix_loop_metrics.py -- not affected -- metrics only, decides nothing from an empty answer

Pattern harvest

Rule candidate: review-prompt
Pattern: an empty result from a filtered search API read as proof of absence instead of "not observed"

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@github-actions

github-actions Bot commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 313b92f25fc6b604aa477f9e3a21208c7a62e98e touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 313b92f25fc6b604aa477f9e3a21208c7a62e98e — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

No design-level concerns: an empty listing now falls back to the head's own check-runs, the non-empty path is unchanged, and a run that cannot be read is reported as unknown, not absent.

Notes

  • The check-runs fallback is now written three times: gh_head_runs in pr-readiness.yml, head_runs_from_check_runs in ai-review-human-override.yml, and _head_runs in republish_withheld_verdict.py. If one copy changes its rule for parsing details_url and the others don't, they will disagree. A test that runs all three on the same fixture would catch that.

[DESIGN-REVIEWED] 313b92f

Description read

[DESCRIPTION-READ] b9fe503639f148c8ae962d5856989da30025503d5f98ccddbf77e360495fb998

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of 313b92f25fc6b604aa477f9e3a21208c7a62e98e — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

The cause has one source (#23245) and the fix works at the mechanism level. Two notes: three deciding readers in .github/** were not swept, and the same fallback is spelled twice in bash.

Notes

  • Item 3 — point patch: the description says "this PR covers the readers that live in .github/**". Grepping head_sha= across .github/ finds 3 deciding readers that the sweep does not list: pr-size-override.yml:164, which prints "PR Policy has not run", sensitive-change-review-comment.yml:71, which prints "No Sensitive Change Review run found", and merge-queue-readiness.yml:193, where latest_run returns null. Subtraction: narrow the description's claim to the three readers this PR changes, and defer the other three.
  • Item 4 — second spelling: head_runs_from_check_runs in ai-review-human-override.yml:152 re-implements the check-runs jq, the per-id fetch and the sort_by(.id) | reverse filter that gh_head_runs already has. Subtraction: delete it and source the shared gh-retry.sh gh_head_runs, as the fork review workflows already do.

What this change ships

Inventory (6 items) — 4 justified

Intent: CI stops reading an empty head_sha runs listing as proof that a head has no runs. This is a FIX for #23245.

  1. PR Readiness lanes on an empty listing are found through the head's check-runs, not shown as "(not started)" — justified
  2. A named run that cannot be read shows "could not be evaluated" — justified
  3. The publish success re-check no longer publishes over a restarted lane hidden by an empty listing — point patch, 3 unswept .github siblings
  4. The override binds the fork Fast Gate and the same-repo lane run through check-runs — second spelling of gh_head_runs
  5. Republish finds the lane run through check-runs, and an unreadable run counts as unknown — justified
  6. ci-and-reviews.md documents the rule — justified

[FIRST-PRINCIPLES-REVIEWED] 313b92f

Description read

[DESCRIPTION-READ] b43b7e010c1e4ca682f48fe3b8ff1c70bad538c4973f2a36a9a053a617598507

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 313b92f25fc6b604aa477f9e3a21208c7a62e98e — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 313b92f

Verdict parsed from the review's SHA-scoped output markers for commit 313b92f25fc6b604aa477f9e3a21208c7a62e98e.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 313b92f25fc6b604aa477f9e3a21208c7a62e98e false-positive: <reason, 20 to 500 characters>

@dodgeblaster
dodgeblaster force-pushed the fix/23245-empty-run-listing branch from 746ab1c to 37c1586 Compare November 2, 2026 18:26
@github-actions

github-actions Bot commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 313b92f25fc6b604aa477f9e3a21208c7a62e98e and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 313b92f

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 313b92f25fc6b604aa477f9e3a21208c7a62e98e false-positive: <reason, 20 to 500 characters>

@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 2, 2026
@dodgeblaster
dodgeblaster force-pushed the fix/23245-empty-run-listing branch from 37c1586 to 0aaf83e Compare November 2, 2026 19:08
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 2, 2026
@dodgeblaster
dodgeblaster force-pushed the fix/23245-empty-run-listing branch from 0aaf83e to dec4407 Compare November 2, 2026 20:06
@dodgeblaster

Copy link
Copy Markdown
Contributor Author
  • Enumerate all check runs before binding to a PR span=de97b0867549

fixed in 0aaf83e: the check-runs fallback reads with filter=all at all three sites (head_runs_from_check_runs, gh_head_runs, _run_ids_from_check_runs), so a sibling PR's newer same-named job on a shared head no longer hides this PR's run. The run bindings are unchanged.
Regression tests serve only the sibling's row to a read without filter=all and assert this PR's run is still found, one per site; each fails without the fix.

@dodgeblaster

Copy link
Copy Markdown
Contributor Author
  • Preserve newest-first ordering for same-repository overrides span=de97b0867549

fixed in dec4407: the check-runs fallback returns its runs newest first (sort_by(.id) | reverse in head_runs_from_check_runs and gh_head_runs; descending ids in _head_runs), the order the listing itself answers in, so rerun_reviewer's first match is the newest run PR Readiness scores.
Regression test test_a_same_repo_empty_listing_re_runs_the_newest_named_lane_run: two named lane runs (321, 400) on one head; the override re-runs 400 and never 321. It fails on the previous head.

@dodgeblaster

Copy link
Copy Markdown
Contributor Author
  • Fallback re-runs the oldest review run instead of the newest span=35540456fe76

fixed in dec4407: the check-runs fallback returns its runs newest first (sort_by(.id) | reverse in head_runs_from_check_runs and gh_head_runs; descending ids in _head_runs), so rerun_reviewer's [0] is the newest run, the one PR Readiness scores with max_by(.id).
Regression test test_a_same_repo_empty_listing_re_runs_the_newest_named_lane_run: two named lane runs (321, 400) on one head; the override re-runs 400 and never 321. It fails on the previous head.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Nov 2, 2026
…runs

The actions/runs?head_sha= listing is a filtered search that has answered
no runs for a head whose own check-runs named 30 of them. PR Readiness,
the AI-review override handler's Fast Gate binding and the withheld-verdict
republisher each took that empty answer to mean the head has no runs, so a
green lane read "(not started)", a fork override was not bound to its
Fast Gate run, and a lane run was "not located".

An empty listing is now read through the head's check-runs: an Actions
job's check-run links its run in details_url, each distinct run is
fetched, and it is kept only when it ran for that event on that exact
head. "No runs" is concluded only when the check-runs name none either,
and a named run that cannot be fetched is a read failure rather than an
absent run. A non-empty listing is used as before, with no extra request.

Fixes #23245
@dodgeblaster
dodgeblaster force-pushed the fix/23245-empty-run-listing branch from dec4407 to 313b92f Compare November 2, 2026 20:56
@dodgeblaster

Copy link
Copy Markdown
Contributor Author
  • A missing dynamic run fans out over every pull_request run span=583f177fc27b

fixed in 313b92f: gh_head_runs takes an optional runs page the caller already holds, and both dynamic reads pass theirs ($pr_runs in the evaluation, $runs_page in the publish re-check). Runs on that page are not fetched again, so a head with no dynamic run costs one check-runs read instead of one read per pull_request run, and those runs can no longer turn a missing dynamic run into "could not be evaluated".
Regression tests test_a_missing_dynamic_run_does_not_refetch_the_known_pull_request_runs (evaluation: only the unknown dynamic run is fetched) and test_no_dynamic_run_does_not_refetch_the_runs_page_in_hand (publish: the CI run on the page is not re-read). Both fail on the previous head.

@dodgeblaster
dodgeblaster marked this pull request as ready for review November 2, 2026 22:33
@dodgeblaster
dodgeblaster requested review from a team and KiroCrewCaptain as code owners November 2, 2026 22:33
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Nov 3, 2026

@iamwhatever iamwhatever left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Request changes at 313b92f: one follow-up needs an issue.

The fix itself looks sound. gh_head_runs and its two siblings use a non-empty listing exactly as before, read an empty one through the head's check-runs, keep only runs for this event on this exact head, and report a named run that cannot be read as unknown rather than absent (pr-readiness.yml L439-L497). Earlier GPT and Opus findings carry dispositions with fix SHAs, and all checks are green.

Blocker: untracked siblings of the same defect. First Principles Item 3 is correct, and nobody has replied to it. Three more .github/ readers take an empty head_sha listing as "no run":

The PR's own sweep (git grep -nE "head_sha=" -- .github/) matches all three, but the Sibling list leaves them out, and the description says this PR covers the readers in .github/**. #23245 does not list them, so merging this closes the only issue that names the defect. Please either apply gh_head_runs (or say why each is safe) here, or open one issue covering the three, link it in the description, and narrow the coverage claim and the Sibling list to match.

AI findings judged non-blocking

  • First Principles Item 4 and the Design note: the check-runs fallback is spelled more than once. Each copy is tested (ai-review-human-override.yml L407); this is a maintainability point with no user-visible harm.

— crew-pr-reviewer (r143)

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention merge conflict Branch has merge conflicts with its base — author must resolve before merge and removed readiness: passed Eligible automated validation passed for the current revision labels Nov 3, 2026
@bolichen97

Copy link
Copy Markdown
Contributor

[operator: bolichen97] Continuing this in #32383; thanks for the start.

bolichen97 added a commit that referenced this pull request Nov 7, 2026
The actions/runs?head_sha= listing is a filtered search that has
answered no runs for a head whose own check-runs named 30 of them.
PR Readiness, the AI review override handler and
republish_withheld_verdict.py took that empty answer as "no runs": a
green lane read as "(not started)", an override was not bound to its
fork lane run, and a republish asked for a manual re-run.

When the listing is empty, each reader now reads the head's check-runs
(every row, filter=all), fetches the runs their Actions job links name,
and keeps those that ran for that event on that exact head, newest
first. A named run that cannot be read is unknown, never absent. A
non-empty listing takes the same path as before.

Builds on #28431.

Co-authored-by: Gary Jennings <garysjennings@gmail.com>
bolichen97 added a commit that referenced this pull request Nov 8, 2026
The actions/runs?head_sha= listing is a filtered search that has
answered no runs for a head whose own check-runs named 30 of them.
PR Readiness, the AI review override handler and
republish_withheld_verdict.py took that empty answer as "no runs": a
green lane read as "(not started)", an override was not bound to its
fork lane run, and a republish asked for a manual re-run.

When the listing is empty, each reader now reads the head's check-runs
(every row, filter=all), fetches the runs their Actions job links name,
and keeps those that ran for that event on that exact head, newest
first. A named run that cannot be read is unknown, never absent. A
non-empty listing takes the same path as before.

Builds on #28431.

Co-authored-by: Gary Jennings <garysjennings@gmail.com>
bolichen97 added a commit that referenced this pull request Nov 8, 2026
The actions/runs?head_sha= listing is a filtered search that has
answered no runs for a head whose own check-runs named 30 of them.
PR Readiness, the AI review override handler and
republish_withheld_verdict.py took that empty answer as "no runs": a
green lane read as "(not started)", an override was not bound to its
fork lane run, and a republish asked for a manual re-run.

When the listing is empty, each reader now reads the head's check-runs
(every row, filter=all), fetches the runs their Actions job links name,
and keeps those that ran for that event on that exact head, newest
first. A named run that cannot be read is unknown, never absent. A
non-empty listing takes the same path as before.

Builds on #28431.

Co-authored-by: Gary Jennings <garysjennings@gmail.com>
@dodgeblaster

Copy link
Copy Markdown
Contributor Author

Closing as superseded: #32383 (@bolichen97, merged 2026-11-08) builds on this PR, landed the same fix on main and closed #23245. The three workflows named in the change request (pr-size-override.yml, sensitive-change-review-comment.yml, merge-queue-readiness.yml) are covered by #33367, tracked in #32348.

@github-actions github-actions Bot removed the readiness: action required A blocking check or review needs attention label Nov 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

merge conflict Branch has merge conflicts with its base — author must resolve before merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

An empty head_sha run listing is read as no runs for the head

3 participants