[BUG]: CI: prior-branch lookup fails when gh run list returns a stale slice missing recent runs

Problem

The prior-branch artifact lookup in ci/tools/lookup-run-id (hardened by #2643) has a known-latent gap that surfaced repeatedly today (2026-09-30): gh run list --branch 12.9.x --workflow CI --status success --limit 100 intermittently returns a bounded candidate set that omits the newest runs. When every returned candidate is older than the 90-day GitHub Actions artifact retention, all lookups fail with:

Error: No successful 'CI' run on branch '12.9.x' has all required artifacts

Ralf explicitly deferred this class in #2643:

This intentionally does not eliminate the possibility that GitHub's server-side branch filter returns a stale or incomplete bounded candidate set; that broader lookup redesign remains outside #2642.

Evidence

The 12.9.x branch has four fresh successful CI runs (all carrying the required cuda-bindings-python*-cuda12.9.1-* artifacts, unexpired):

$ gh api "repos/NVIDIA/cuda-python/actions/workflows/ci.yml/runs?branch=12.9.x&status=success&per_page=5" \
    --jq '.workflow_runs[] | [.id, .created_at]'
[35797318627, "2026-09-22T23:25:12Z"]
[35362127153, "2026-09-18T15:23:32Z"]
[35166820782, "2026-09-17T00:30:14Z"]
[34881968601, "2026-09-14T18:37:49Z"]
[31739833756, "2026-08-13T20:14:11Z"]

Three CI jobs today independently invoked lookup-run-id and each received a different stale slice, none containing any 2026-09 run:

Job	Time (UTC)	Newest run in slice	Slice size	Result
#2920, `aarch64 / py3.11` build	14:29:04	22911991598 (2026-03-10)	12	fail
#2965, `linux-64 / py3.10` build	17:04:06	23885913601 (2026-04-02)	13	fail
#2965, `linux-64 / py3.14t` test	17:24:02	26890119460 (2026-06-03)	14	fail

Sibling matrix rows in the same runs made the same lookup within seconds and mostly got fresher slices, so this manifests as a per-row flake. #2965's py3.10 build passed on run_attempt: 2 with no code change, confirming the flake is API-side.

Root cause

gh run list --branch --workflow --status success --limit 100 occasionally returns a bounded set that omits the true newest runs on the branch. The tool trusts this list as authoritative; when the entire returned slice is older than the 90-day artifact retention, every candidate fails the artifact-freshness check and the step errors out. This is the residual API-caching risk Ralf called out in #2643.

Possible mitigations

Cross-check gh run list against a direct REST call to /repos/{r}/actions/workflows/ci.yml/runs?branch=X&status=success&per_page=100, union the results, then sort. A local replay of this endpoint today consistently returned all 2026-09 runs even while the CI's gh run list did not.
Look up the branch HEAD SHA (/repos/{r}/branches/12.9.x) first and try the run for that SHA before falling back to the list.
Retry gh run list once with a short delay after exhausting candidates; the stale slice usually clears on a second call.
On final failure, emit a diagnostic that dumps the returned slice and the branch HEAD, so a reviewer doesn't have to reproduce the API state to see what happened.

-- Leo's bot

@jpascucci-nv

@jpascucci-nv would you be able to help take a look? 🙂

@leofang

@leofang Gemini has a different solution.

This is caused by a known server-side caching issue with the GitHub API when combining the status=success filter with branch/workflow constraints.
When you pass --status success, the GitHub server processes the status filter first on its end. The endpoint frequently returns a stale or intermittently cached response dataset, omitting the absolute latest runs.
The Solution: Filter Locally
To get the true latest successful runs, omit the --status flag so GitHub returns the absolute newest runs sequentially, and then use jq to isolate the successful ones.
Run this command instead:
bash
gh run list --branch <branch_name> --workflow <workflow_name> --limit 100 --json conclusion,status,databaseId,createdAt,displayTitle --jq '.[] | select(.conclusion == "success")'
Why this works:
• Bypasses Buggy API Cache: Removing --status success forces the API to pull the most recent runs directly from the live database.
• --json and --jq: The --json flag fetches the structured data fields, and the jq selector mimics the exact filter behavior locally without data loss.
If you'd like to narrow this down or build a script, let me know what specific fields (like the run ID, number, or commit message) you need to extract from those latest runs

(links to github issues can be found by clicking the above link, the copy-paste dropped them)

This shouldn't. have been closed.

Closes NVIDIA#2975 (NVIDIA#2976)

It looks like the fix didn't work, there was a nightly build job which failed with https://github.com/NVIDIA/cuda-python/actions/runs/37013218624/job/110862010965 . I'll try one of the alternate methods.

Not related to lookup-run-id but same class of errors: I manually triggered the nightly-optional-deps CI and saw that it fetches from a terribly old commit (1db44ec is from two weeks ago):
https://github.com/NVIDIA/cuda-python/actions/runs/37046383242/job/110968754150#step:3:39

@leofang

Replying to @leofang's note above — this one actually is lookup-run-id. Step 3 of find-wheels is literally this call (log line 172 in the step):

OUTPUT=$(./ci/tools/lookup-run-id --branch main --head-sha "NVIDIA/cuda-python" "CI")

The step didn't error — it reported Found run ID: 35686532745 and exited 0 — but that run is from 2026-09-22T04:20Z. At the time find-wheels ran (2026-10-02T18:17Z) the true newest successful CI run on main was 37013218624 (2026-10-02T13:28Z), and four more fresher successes also existed (36822…, 36917…, 36931…, 36938…). None of them appeared in the slice gh run list returned, so the first candidate was the Sep 22 one — and branch-mode in lookup-run-id accepts the first candidate unconditionally when no --artifact is passed, so it was returned.

So the bug is the same as before, with --branch main doing the work --status success used to do: community#24626 explicitly lists branch, status, and created as all affected by the stale-slice behavior. #2976 dropping --status likely reduced the flake rate but didn't close it. @jpascucci-nv's earlier example (runs/37013218624/job/110862010965, 2026-10-02T13:47Z) is the same tool in the loud failure mode — the test-wheel-windows path passes --artifact, so it iterates candidates, exhausts the stale slice, and errors.

Worth noting the failure mode is now silent when no --artifact is passed: before #2976 the typical signature was a loud Error: No successful 'CI' run on branch has all required artifacts; now the tool returns a stale-but-unexpired run and downstream jobs break far away. Four of today's failing nightly jobs die at Display structure of downloaded cuda-python artifacts with ls: cannot access 'cuda_python*.whl': No such file or directory (e.g. this one) because the Sep 22 run's artifact set doesn't match today's test code.

Mitigations, roughly cheapest first:

Resolve the branch HEAD (gh api repos/{r}/branches/main) first and try the run for that SHA; use the list query only as a fallback when HEAD has no successful run yet.
Drop --branch from gh run list too (keep only --workflow), filter branch client-side — matches 24626's "unfiltered + client-side" workaround (10/10 correct in the thread).
Retry gh run list once on a short backoff if the top candidate is older than N hours — the stale slice usually clears on a second call.
In branch mode, always run the sanity-check loop even when --artifact isn't passed (e.g. require at least a HEAD-SHA-recent-ness check), so a silently-stale pick becomes a loud skip.

-- Leo's bot

ci: resolve branch run by walking HEAD history (#3013)

#3013 was merged. Let's keep an eye on this.

Our admins opened a support ticket w/ GH (4817893). GH Support noted that this issue should be mitigated now. I have my bot running a sweep to see if the same error still happens since #3013 was merged. Though even if there is 0 error that happened since then, it's hard to say whether it's our #3013 fix or GH's fix taking effect, in particular considering we basically (almost) follow the same suggestion that GH provided:

The reliable workaround is to drop every server-side filter and apply the conditions in your own client instead. On the REST endpoint that means leaving off branch, status, event, created and actor, and on gh run list it means leaving off --branch, --status, --event, --created, --user and --commit, since all of them go through the same search path. This is also why dropping --status alone did not hold up for you in #2975. Retrying the same filtered request is not a dependable safeguard on its own either, because the incomplete responses tend to recur within the same burst.

have my bot running a sweep to see if the same error still happens since #3013 was merged.

doesn't seem to find anything unusual, let's close this for now

Navigation Menu

Repository navigation

[BUG]: CI: prior-branch lookup fails when `gh run list` returns a stale slice missing recent runs #2975

Problem

Evidence

Root cause

Possible mitigations

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[BUG]: CI: prior-branch lookup fails when gh run list returns a stale slice missing recent runs #2975

Description

Problem

Evidence

Root cause

Possible mitigations

Activity

leofang commented on Sep 30, 2026

jpascucci-nv commented on Sep 30, 2026

jpascucci-nv commented on Sep 30, 2026

jpascucci-nv commented on Oct 2, 2026

leofang commented on Oct 2, 2026

leofang commented on Oct 2, 2026

leofang commented on Oct 6, 2026

leofang commented on Oct 9, 2026

leofang commented on Oct 9, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

[BUG]: CI: prior-branch lookup fails when `gh run list` returns a stale slice missing recent runs #2975