Skip to content

fix: read an empty head_sha run listing through check-runs - #32383

Merged
bolichen97 merged 1 commit into
mainfrom
fix/23245-check-runs-fallback
Nov 8, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
fix/23245-check-runs-fallback

Conversation

@bolichen97

@bolichen97 bolichen97 commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: CI stops treating an empty actions/runs?head_sha= answer as proof that a head has no runs, so a green lane is not read as "(not started)" and a lane run is not reported as missing when it exists.

The head_sha-filtered runs listing is a search that has answered total_count: 0 for a head whose own check-runs named 30 runs (#23245). The three .github/** readers #23245 names took that empty answer at face value:

  • PR Readiness read every lane as (not started) and held the verdict at pending; its success re-check found no open lane and could publish success over a lane that had just restarted.
  • ai-review-human-override.yml reported "Fast Gate has not run" on a fork PR (so the override was not bound to the fork lane run), and "no workflow run found" on a same-repo PR.
  • republish_withheld_verdict.py reported the lane run "could not be located" and asked for a manual re-run.

The pr_status.py reader was fixed in #23253. Builds on #28431 by @dodgeblaster.

Why it matters

The required PR Readiness status can sit at pending on a green PR until a later recompute happens to get a non-empty answer. An override or republish that cannot find its run leaves a lane blocked or unpublished and asks a person to re-run it by hand. Each of these happens on a head whose runs exist.

Not a goal

  • Changing why GitHub's filtered listing comes back empty, or reading runs from a different primary source. The listing is still the first read.
  • await-fast-gate.sh, which polls its listing every 10s and re-reads an intermittent empty answer.
  • Display-only readers (fix_loop_metrics.py, Issue Radar list_pr_workflow_runs), which decide nothing from an empty answer.
  • A run that has no job check-run yet (a fork run awaiting approval, a run queued behind a concurrency group) stays invisible to the fallback until the listing answers again; the fallback reads only the runs the head's check-runs name.
  • The three readers An empty head_sha run listing is read as no runs for the head #23245 does not name (pr-size-override.yml, sensitive-change-review-comment.yml, merge-queue-readiness.yml): tracked in Three more workflows read an empty head_sha run listing as no run #32348.

What changed (motivation → approach → change)

Symptom: an empty listing reads as "no runs". Root cause: each reader has only one source for a head's runs. Change: when the listing is empty, read the head's check-runs. An Actions job's check-run links its run in details_url (.../actions/runs/<run id>/job/<job id>). Each distinct run is fetched and kept only when it ran for that event on that exact head. This is the same idea as #23253: the head's own check data outranks an empty listing.

Reader Empty listing reads through A named run that cannot be read
pr-readiness.yml evaluation and publish re-check gh_head_runs in the shared gh-retry.sh helper could not be evaluated, never (not started)
ai-review-human-override.yml fork Fast Gate binding and same-repo lane lookup head_runs_from_check_runs a read-failure fail_lane, never "has not run" or "not found"
republish_withheld_verdict.py _same_repo_run_id and _fork_trigger _head_runs None (unknown)

The check-runs read asks for every row (filter=all), so a sibling PR's newer same-named job on a shared head does not hide this PR's run, and the runs come back newest first, the order the listing answers in. A non-empty listing is used exactly as before and makes no extra request. An empty listing costs one check-runs read, plus one read per run it names that the caller does not already hold: the dynamic (CodeQL) reads skip the runs already on the pull_request page, so a head with no dynamic run costs one check-runs read. docs/ci/ci-and-reviews.md documents the rule.

Backwards compatibility

Compatible: a non-empty listing takes the same path as on main, and an empty listing with no Actions job check-runs still reads as "no runs".

Removes nothing: the change only adds a second source behind an empty answer. No path, field, validator or record is taken away.

Tests

  • test_pr_readiness_empty_run_listing.py: runs named by the check-runs answer for an empty listing, also when a sibling's newer same-named row is all the default filter returns (verdict success). A named run that cannot be read, or a check-runs read that fails, gives could not be evaluated. A named run for another head or event is not borrowed. An empty listing with no job check-runs is still (not started). A non-empty listing reads no check-runs and fetches no single run. A missing dynamic run does not re-fetch the pull_request runs in hand. The module skips only off POSIX (no bash, Windows); a host without jq fails its runner fixture with a named error instead of skipping, because the fallback under test is a jq program.
  • test_pr_readiness_publish.py: a lane restarted behind an empty listing still holds the success. An unreadable named run does not publish success. A missing dynamic run does not re-fetch the runs page in hand.
  • test_ai_review_workflows_human_override.py: fork and same-repo, an empty listing binds or re-runs the run the check-runs name, the newest when two are named. An unreadable named run, or a failed check-runs read, is a read failure. A run for another head or event does not bind. No job check-runs still reads "has not run" / "not found". The .details_url guard is scoped to the fork lane resolver it protects, because that resolver still reads the lane run only from the marker.
  • test_republish_withheld_verdict.py: fork and same-repo lookups through the check-runs; the no-readable-run cases, where an unreadable run or check-runs read is unknown (None) and a run for another head or event is absent ([]); no check-runs read on a non-empty listing; run ids parsed from Actions job links across every page; and the check-runs read asking for every row (filter=all).

Determinism: the 28 new test ids pass 20 times at -n0, 10 times at -n 4 --dist loadgroup beside test_pr_readiness_codeql_sources.py and test_pr_readiness_fork_base.py, and in a shuffled order (seed 17705).

Each read-failure, unknown-versus-absent and event-filter test above fails with its own guard reverted by hand.

Proof: not provable -- prove.py timed out

Manual verification

N/A — every changed reader is a workflow step or script that the tests execute against a stubbed gh. The intermittent empty listing cannot be produced on demand against the live API.

Screenshots / video

N/A — CI workflows and scripts only; no user-visible surface.

Related Issues

Closes #23245

Sibling sweep

Swept: git grep -nE "actions/(workflows/[^/]+/)?runs" -- .github/ scripts/ src/ (kept: the reads filtered on head_sha)
Sibling: .github/workflows/pr-readiness.yml -- changed here
Sibling: .github/workflows/ai-review-human-override.yml -- changed here
Sibling: .github/scripts/republish_withheld_verdict.py -- changed here
Sibling: .github/workflows/pr-size-override.yml -- tracked in #32348
Sibling: .github/workflows/sensitive-change-review-comment.yml -- tracked in #32348
Sibling: .github/workflows/merge-queue-readiness.yml -- tracked in #32348
Sibling: src/kiro_crew/builtin_skills/kirocrew-dev/kirocrew-prepare-pr/scripts/pr_status.py -- not affected -- fixed by the rollup-row check in #23253
Sibling: .github/scripts/await-fast-gate.sh -- not affected -- it polls every 10s and re-reads an intermittent empty answer
Sibling: .github/scripts/fix_loop_metrics.py -- not affected -- metrics only, decides nothing from an empty answer
Sibling: src/kiro_crew/apps/builtins/issue_radar/backend/github_client_writes.py -- not affected -- display only, decides nothing from an empty answer
Sibling: src/kiro_crew/builtin_skills/kirocrew-dev/kirocrew-prepare-pr/SKILL.md -- not affected -- its cancel snippet reads in-flight runs to cancel; an empty answer cancels nothing

Pattern harvest

Rule candidate: review-prompt
Pattern: an empty result from a filtered search API read as proof of absence instead of "not observed"

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@bolichen97
bolichen97 requested review from a team and KiroCrewCaptain as code owners November 7, 2026 14:30
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 7, 2026
@github-actions

github-actions Bot commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 93c191b04bfe1cad47a59c1fd3e14c324d9df3a9 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 93c191b04bfe1cad47a59c1fd3e14c324d9df3a9 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: the fix targets the cause (each reader trusted a single source), it only adds a fallback behind an empty listing, and an unreadable named run reads as unknown rather than absent.

Notes

  • The same fallback now exists three times: gh_head_runs (bash, pr-readiness.yml), head_runs_from_check_runs (bash, ai-review-human-override.yml) and _head_runs (Python, republish_withheld_verdict.py). Each copy has its own details_url parsing, event/sha filter and newest-first order. Before Three more workflows read an empty head_sha run listing as no run #32348 adds three more readers, make one shared script that all of them call, or the copies will drift apart on how they read the filter=all answer.

[DESIGN-REVIEWED] 93c191b

Description read

[DESCRIPTION-READ] df6594c2658a84d57fca928289935fb84958d11213f91e557bceaabf89896729

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of 93c191b04bfe1cad47a59c1fd3e14c324d9df3a9 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Nothing blocks. Each reader fixes the cause where it sits: an empty head_sha listing was its only source for a head's runs. The three readers left out are tracked in #32348.

Inventory (8 items) — 8 justified

Intent: CI stops reading a green or running lane as missing when GitHub's filtered runs listing comes back empty for a head whose runs exist. This is a FIX (#23245).

  1. PR Readiness reads an empty pull_request runs listing through the head's check-runs instead of marking lanes "(not started)" — justified
  2. PR Readiness's CodeQL (dynamic) read gets the same fallback and skips runs already fetched — justified
  3. The publish step's success re-check sees a lane that restarted behind an empty listing — justified
  4. A named run that cannot be read gives "could not be evaluated", never "(not started)" — justified
  5. The override workflow's fork Fast Gate binding finds the lane run through check-runs — justified
  6. The override workflow's same-repo lane lookup finds the run instead of "no workflow run found" — justified
  7. The republish script finds the lane run; an unreadable run counts as unknown, not absent — justified
  8. ci-and-reviews.md states the "empty listing is not no runs" rule — justified

[FIRST-PRINCIPLES-REVIEWED] 93c191b

Description read

[DESCRIPTION-READ] 4d84ce4e699e5c7d681fbf4dd4f4c887eb7cab7ef064c9baee4ad71eb50d7a83

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 93c191b04bfe1cad47a59c1fd3e14c324d9df3a9 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 93c191b

Verdict parsed from the review's SHA-scoped output markers for commit 93c191b04bfe1cad47a59c1fd3e14c324d9df3a9.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 93c191b04bfe1cad47a59c1fd3e14c324d9df3a9 false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 93c191b04bfe1cad47a59c1fd3e14c324d9df3a9 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 93c191b

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 93c191b04bfe1cad47a59c1fd3e14c324d9df3a9 false-positive: <reason, 20 to 500 characters>

@bolichen97
bolichen97 enabled auto-merge (squash) November 7, 2026 14:44
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 7, 2026
@bolichen97

bolichen97 commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor Author
  • span=05fc40eb8d6a Imported marker skips all new fallback tests when jq is absent

Rebutted: disproportional. jq is the interpreter of the subject here, not a tool the harness can stand in for. The seven tests run the real "Evaluate current revision" step, and the fallback they pin is written in jq: the jq -cs page merge, the jq -rs read of run ids out of each check-run details_url, and the jq -cs rebuild of the runs listing.
A stand-in that answers those programs would reimplement them (the step runs jq on 23 lines), and the tests would then check the stand-in instead of the workflow. The supplied-jq case in testing-conventions.md covers three fixed calls (-Rs ., -r '.title', -r '.body // ""') around a step whose logic is shell, not a jq program under test.
The skip is the evaluate harness's own pytestmark (test/test_pr_readiness_evaluate.py:39-45), imported unchanged with that harness. Its predicate checks only the host tools and cannot fire on the subject failing, which is the rule's allowance for a test this diff adds.
Dropping the jq term without a stand-in turns the skip into a red on a jq-less host that says nothing about the subject. On CI the seven tests run: New-test stability (ubuntu-latest) reports 320 passed, 0 skipped, on three passes at this head.
[operator: bolichen97]

@bolichen97

Copy link
Copy Markdown
Contributor Author

self-added: yes
mechanism: the new module imported the evaluate harness's bash + jq skip marker

  • Imported marker skips all new fallback tests when jq is absent span=05fc40eb8d6a

fixed in 9cd2f5e: the module defines its own marker that skips only off POSIX (no workflow, Windows, no bash). A missing jq is no longer a skip: the runner fixture calls pytest.fail naming jq, since the fallback under test is a jq program no stand-in could check.
Evidence: on a PATH with no jq the seven tests went from 7 skipped (parent) to 7 errors with the named message; with jq they pass (7 passed).

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Nov 7, 2026
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 8, 2026
The actions/runs?head_sha= listing is a filtered search that has
answered no runs for a head whose own check-runs named 30 of them.
PR Readiness, the AI review override handler and
republish_withheld_verdict.py took that empty answer as "no runs": a
green lane read as "(not started)", an override was not bound to its
fork lane run, and a republish asked for a manual re-run.

When the listing is empty, each reader now reads the head's check-runs
(every row, filter=all), fetches the runs their Actions job links name,
and keeps those that ran for that event on that exact head, newest
first. A named run that cannot be read is unknown, never absent. A
non-empty listing takes the same path as before.

Builds on #28431.

Co-authored-by: Gary Jennings <garysjennings@gmail.com>
@bolichen97
bolichen97 force-pushed the fix/23245-check-runs-fallback branch from 9cd2f5e to 93c191b Compare November 8, 2026 01:25
@bolichen97

Copy link
Copy Markdown
Contributor Author

[operator: bolichen97] Rebased onto main dcf5084, which carries #32786. Backend shard 6 was red twice on the old base: test/test_steer_retract.py::TestAHardStop::test_a_drop_during_discarded_delivery_settle_cannot_requeue (stub missing published_queued_labels, fixed by #32786), and before that flake #32796. Diff unchanged; CI re-running.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Nov 8, 2026

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving head 93c191b. The review read the full diff against the surrounding code. The changed suites passed at this head: 36 tests in the empty-listing and republish files, and 284 in the publish, human-override and evaluate files. A mutation pass reverted each new guard in turn, and every one turned at least one test red: the fallback, the event and head_sha filter, filter=all, the known-run skip, the newest-first sort, and fail-closed on an unreadable run. Nothing on main contains this change yet.

Security: this changes how the required PR Readiness gate and the human override workflow find lane runs. The risk is not realized. A run id read from a check-run's details_url only selects a run that is then fetched from this repository's own Actions API. That run is kept only when its event and head_sha match, and it then goes through the same head repository and branch binding as a listing row. A forged link can only name a real run of this repository on this exact head. Every read failure is reported as "unknown", never as "absent": readiness says "could not be evaluated", the override fails the lane, and republish returns None.

Done well: each reader keeps "unknown" separate from "absent", and a separate test covers each of those cases.

Optional:

  • test/test_ai_review_workflows_human_override.py:543 now checks for .details_url only inside rerun_fork_lane(). You could drop just the head_runs_from_check_runs() body from the script and assert on the rest. That is stricter and still passes at this head.
  • The body says a head with no dynamic run costs one check-runs read. Runs from other events on the head (for example pull_request_target) are each still fetched once before the event filter drops them.
  • The same-repo override path runs the check-runs fallback once per lane. Caching it once per step, as bind_fork_trigger does, would save REST calls on /ai-review override all.

After this merges, #28431 can be closed as superseded, with credit to its original author.

@bolichen97
bolichen97 merged commit c7a1602 into main Nov 8, 2026
97 checks passed
@bolichen97
bolichen97 deleted the fix/23245-check-runs-fallback branch November 8, 2026 03:19
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Nov 8, 2026
bolichen97 added a commit that referenced this pull request Nov 9, 2026
The actions/runs head_sha listing is a filtered search that can answer
no runs for a head whose runs exist. #32383 made PR Readiness, the AI
review override handler and the withheld-verdict republisher read that
empty answer through the head's check-runs. Three readers still took it
as "no run": a PR Size approval was recorded but PR Policy was not
re-run, a Sensitive Change Review author comment failed with "No
Sensitive Change Review run found", and the merge-queue poll would fail
a green group once its five-minute appear window passed.

Each now reads an empty listing through the head's check-runs (every
row, filter=all), fetches the runs their Actions job links name, and
keeps those of its own workflow and event on that exact head. A named
run that cannot be read is a read failure, never an absent run. The
four workflows involved gain checks: read for that read.
bolichen97 added a commit that referenced this pull request Nov 9, 2026
The actions/runs head_sha listing is a filtered search that can answer
no runs for a head whose runs exist. #32383 made PR Readiness, the AI
review override handler and the withheld-verdict republisher read that
empty answer through the head's check-runs. Three readers still took it
as "no run": a PR Size approval was recorded but PR Policy was not
re-run, a Sensitive Change Review author comment failed with "No
Sensitive Change Review run found", and the merge-queue poll would fail
a green group once its five-minute appear window passed.

Each now reads an empty listing through the head's check-runs (every
row, filter=all), fetches the runs their Actions job links name, and
keeps those of its own workflow and event on that exact head. A named
run that cannot be read is a read failure, never an absent run. The
four workflows involved gain checks: read for that read.
bolichen97 added a commit that referenced this pull request Nov 9, 2026
…33367)

The actions/runs head_sha listing is a filtered search that can answer
no runs for a head whose runs exist. #32383 made PR Readiness, the AI
review override handler and the withheld-verdict republisher read that
empty answer through the head's check-runs. Three readers still took it
as "no run": a PR Size approval was recorded but PR Policy was not
re-run, a Sensitive Change Review author comment failed with "No
Sensitive Change Review run found", and the merge-queue poll would fail
a green group once its five-minute appear window passed.

Each now reads an empty listing through the head's check-runs (every
row, filter=all), fetches the runs their Actions job links name, and
keeps those of its own workflow and event on that exact head. A named
run that cannot be read is a read failure, never an absent run. The
four workflows involved gain checks: read for that read.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

An empty head_sha run listing is read as no runs for the head

2 participants