Skip to content

fix(ci): wait out a spent rate limit in security scope review reads - #33680

Merged
KiroCrewCaptain merged 1 commit into
mainfrom
fix/14382-scope-read-rate-limit
Nov 11, 2026
Merged

KiroCrewCaptain merged 1 commit into
mainfrom
fix/14382-scope-read-rate-limit

Conversation

@bolichen97

@bolichen97 bolichen97 commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A same-repo pull request whose Security Scope Review run meets a spent GitHub API rate limit that resets within 20 minutes gets a measured verdict after the reset, instead of a red head that no re-run can clear.

On a same-repo pull request, Security Scope Review reads the PR's comments (to find a human override) and the head's earlier check-runs (for the per-head floor). Each read tried three times within about three seconds. When the shared installation rate limit was spent, all three tries failed, the run failed closed, and that failure became the head's floor: a later re-run that measured success was held at it. This happened on #13647 (the comment read), #13335 and #32110's PR #30937 (the check-run read).

Why it matters

Every same-repo author whose push lands while the shared token is drained gets a red Security Scope Review that a re-run cannot clear. Today the only ways out are a new push, which re-rolls every other lane and often meets the same limit, or a writer's /ai-review override scope <sha>.

Not a goal

  • Changing what the per-head floor decides, or letting the same-repo lane mark its own run unsettled. The strict floor is a recorded ruling and stays as it is.
  • A rate-limit wait on any other call in this lane, or in any other lane.
  • A reset more than 20 minutes away: that run still fails closed, and the head takes /ai-review override scope <sha>, as does a head a past flake already floored.

What changed (motivation → approach → change)

A same-repo pull request whose Security Scope Review run meets a drained API token is no longer left red for good when the limit resets within 20 minutes: the run waits for the reset and then measures the change. The PR-comment read and the earlier check-runs read used to give up after about three seconds, but a spent installation rate limit resets within the hour, so the run failed closed and the per-head floor kept that failure.

Both reads wait for the reset when a failed try's error names a rate limit. They read the REST limit's reset time from gh api rate_limit (which costs no quota), sleep until 5 seconds past it (60 seconds when the REST limit is not spent, which means a secondary limit), and try again. Each read waits at most 20 minutes in all, the same budget .github/actions/gh-rate-limit and the fork Opus lane use; a reset further away than that fails closed exactly as before. Any other error keeps the old short retry. The wait is written inline in each step because generate runs no repository program while it holds the Bedrock credential.

The generate job's timeout goes from 30 to 50 minutes and the publish job's from 20 to 40, so the wait fits inside the job. Security Scope Review's fail-closed ruling in docs/ci/ci-and-reviews.md says the two reads wait, and that the wait never changes what the floor decides.

Backwards compatibility

Compatible: a read that succeeds, or fails for any reason other than a rate limit, behaves exactly as before; a rate-limited read waits and then gives the same answer or the same fail-closed error.

Removes nothing: no check, output or verdict is dropped; the floor's inputs and its decision are unchanged.

Tests

  • test/test_ai_review_workflows_human_override.py::TestOverrideReadFailureFailsClosed::test_scope_lane_waits_out_a_spent_rate_limit runs the real override read with gh, date and sleep stubbed: a rate-limited first try waits 105 s (reset 100 s away, plus 5) and then resolves the record; a reset past the 20-minute budget gets no long wait and still fails closed after three tries.
  • test/test_ai_review_workflows_scope_verdict.py::TestPerHeadMonotonicFloor::test_same_repo_floor_read_waits_out_a_spent_rate_limit runs the real floor step: a rate-limited first listing read waits 105 s, then the floor settles on the listing and the step passes.

Proof: PROVEN test/test_ai_review_workflows_human_override.py::TestOverrideReadFailureFailsClosed::test_scope_lane_waits_out_a_spent_rate_limit[1100-1-0-0-105\n-xx], test/test_ai_review_workflows_scope_verdict.py::TestPerHeadMonotonicFloor::test_same_repo_floor_read_waits_out_a_spent_rate_limit (head 5004346)

Manual verification

Both edited step scripts pass bash -n. Not run against a live spent limit, which cannot be produced on demand; the tests above execute the two real step scripts with gh, date and sleep stubbed.

Screenshots / video

N/A: no user-visible surface changes; this is a CI workflow, its tests and one docs paragraph.

Related Issues

Refs #14382

Sibling sweep

Swept: git grep -nE "for attempt in" -- .github/workflows/security-scope-review.yml .github/workflows/fork-security-scope-review.yml
Sibling: .github/workflows/security-scope-review.yml:2449 -- not affected -- the verdict post's comment reads end their step with exit 0 and keep an unposted verdict as an artifact, so they never fail the run
Sibling: .github/workflows/fork-security-scope-review.yml:1850 -- not affected -- a fork run that cannot read its priors marks its own check-run unsettled and sets no per-head floor, and a failed fork override read (:316) warns and reviews normally

Pattern harvest

Not generalizable: the hazard is specific to a gate whose own failed run becomes its next input (the same-repo per-head floor); other lanes' failed runs are cleared by a re-run.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@github-actions

github-actions Bot commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 5004346fd993b902c8e2225968f1df5f05c62f66 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 5004346fd993b902c8e2225968f1df5f05c62f66 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: the fix stays inside the two reads whose failure becomes the head's floor, and the floor's decision is unchanged.

Notes

  • wait_out_rate_limit is now copied into two steps. Only generate has to keep it inline, because it holds the Bedrock credential. The publish copy could later move to a shared script so the two waits can't drift apart.

[DESIGN-REVIEWED] 5004346

Description read

[DESCRIPTION-READ] 9618bc86aef6582852951c35116ec60185779532b5ed4ee45b82f839f1d75e60

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of 5004346fd993b902c8e2225968f1df5f05c62f66 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

This fix aims at the cause: a read that hits a spent rate limit gives up before the reset and floors the head. One copy of the wait duplicates an existing repo mechanism, which is a note, not a block.

Notes

  • Item 2 — duplicate of .github/actions/gh-rate-limit: generate has to carry the wait inline because it "runs NO repository program at all" (security-scope-review.yml:60). publish has no such rule; it checks out and runs repository scripts. So its copy is a third spelling: the action plus two copies of wait_out_rate_limit() in this diff. The action would wrap every gh call in publish, which is outside ## Not a goal, so this stays a note. Subtraction: if the Goal is ever widened, delete the publish copy of wait_out_rate_limit() and use the action there.

What this change ships

Inventory (7 items) — 6 justified

Intent: stop a same-repo Security Scope Review head going permanently red because the shared API limit was spent when it ran. FIX.

  1. The override's PR-comment read waits up to 20 min for a rate-limit reset instead of failing within ~3 s — justified
  2. The per-head floor's read of prior check-runs waits the same way — duplicate of .github/actions/gh-rate-limit, an inline second copy
  3. generate job timeout raised from 30 to 50 min — justified
  4. publish job timeout raised from 20 to 40 min — justified
  5. Both reads now save gh's error output to a temp file and print it again (needed to detect a rate limit) — justified
  6. The fail-closed paragraph in ci-and-reviews.md now says both reads wait — justified
  7. Tests that stub gh, date and sleep to check the 105 s wait and the over-budget case — justified

[FIRST-PRINCIPLES-REVIEWED] 5004346

Description read

[DESCRIPTION-READ] a42fc9eca415369b0bfde8002d8a5dcf81730b09818ea8c5547bb9fc609e3cfd

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 5004346fd993b902c8e2225968f1df5f05c62f66 — this comment is updated in place on each push.

Review details

No findings.

Notes:

  • docs/ci/ci-and-reviews.md says a reset past the 20-minute budget "fails closed at once". The code still runs its two quick 1s/2s retries first, as the test case pins, so "at once" is loose wording rather than a behaviour defect.

[OPUS-REVIEWED] 5004346

Verdict parsed from the review's SHA-scoped output markers for commit 5004346fd993b902c8e2225968f1df5f05c62f66.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 5004346fd993b902c8e2225968f1df5f05c62f66 false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 5004346fd993b902c8e2225968f1df5f05c62f66 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 5004346

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 5004346fd993b902c8e2225968f1df5f05c62f66 false-positive: <reason, 20 to 500 characters>

@bolichen97

Copy link
Copy Markdown
Contributor Author

[operator: bolichen97] Bundle Size Gate is red here because of main, not this diff (it touches no frontend file): the App chunk is over its ceiling on main, tracked in #33668, with fixes open in #33662, #33665 and #33669. This PR waits for one of those to merge, then rebases onto it. The earlier red review lanes on this head were installation rate-limit casualties (reset 04:13:28Z) and were re-run once after the reset.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 9, 2026
On a same-repo pull request, Security Scope Review read the PR comments
(for a human override) and the head's earlier check-runs (for the
per-head floor) three times within three seconds. A spent installation
rate limit resets within the hour, so all three tries failed, the run
failed closed, and that failure became the head's floor: a later
re-run that measured success was held at it.

Both reads now wait for the limit's reset when it is within 20
minutes, the budget the shared gh-rate-limit action uses; a later reset
still fails closed at once. What the floor decides does not change.
The two jobs' timeouts grow by the wait.

Refs #14382
@bolichen97
bolichen97 force-pushed the fix/14382-scope-read-rate-limit branch from 06d13a9 to 5004346 Compare November 9, 2026 23:35
@bolichen97

Copy link
Copy Markdown
Contributor Author

Rebased: main moved in docs/ci/ci-and-reviews.md and in workflows beside .github/workflows/security-scope-review.yml since this head's CI ran (2caa69b -> 93ca1d9); CI re-running.

The new base carries the Bundle Size Gate fix (#33665) this head was waiting on. The same push fixes this PR's two new rate-limit tests on the Windows shards: they now stub sleep and date as shell functions, because Git for Windows' bash puts its own /usr/bin ahead of PATH, where the real commands beat a stub file.

[operator: bolichen97]

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Nov 9, 2026

@buluoray buluoray left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve on 5004346. The fail-closed gate is not weakened: the change only adds a bounded wait before re-reading, a read that never succeeds still exits 1 and the floor's inputs and decision are unchanged. The Windows test failures that blocked the prior head (06d13a9) are fixed by turning the sleep/date PATH stubs into shell functions, and every Windows lane is green on this head. Decisive lines: .github/workflows/security-scope-review.yml:307-335 (override read: wait only on a "rate limit" stderr, 1200 s cap, read_ok still set only by a successful read, exit 1 after 3 failures unchanged), .github/workflows/security-scope-review.yml:3009-3037 (floor read: same wait, readable="false" path and floor decision at :3046-3060 unchanged), test/test_ai_review_workflows_human_override.py (sleep/date as shell functions), test/test_ai_review_workflows_scope_verdict.py (same)

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Nov 10, 2026
@bolichen97 bolichen97 closed this Nov 10, 2026
auto-merge was automatically disabled November 10, 2026 17:40

Pull request was closed

@bolichen97 bolichen97 reopened this Nov 10, 2026
@bolichen97

Copy link
Copy Markdown
Contributor Author

[operator: bolichen97] Closed and reopened with no code change: PR Readiness was waiting on a Code Review run that records its base (the #33337 rule), and only a new pull request event starts one. The close turned auto-merge off; it is re-armed once this re-run is green.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 10, 2026
@bolichen97 bolichen97 closed this Nov 11, 2026
@bolichen97 bolichen97 reopened this Nov 11, 2026
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Nov 11, 2026
@bolichen97

Copy link
Copy Markdown
Contributor Author

[operator: bolichen97] Re-ran CI on a fresh merge ref by closing and reopening this PR; nothing was pushed and the diff is unchanged.

The last run's only reds were test/test_session_markdown.py::test_a_condition_one_opener_is_closed_as_the_parser_reads_it[long-s] on Backend Tests shard 4 (Linux and Windows) and the Coverage Gate that follows from it. That node id was red on main at the same time (markdown-it-py 4.3.0), and #34646 fixed it in 4477b72. The previous merge ref predated that fix; the new one (6c3a776) carries it.

@bolichen97 bolichen97 closed this Nov 11, 2026
@bolichen97 bolichen97 reopened this Nov 11, 2026
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Nov 11, 2026
@KiroCrewCaptain
KiroCrewCaptain merged commit c246059 into main Nov 11, 2026
321 of 362 checks passed
@KiroCrewCaptain
KiroCrewCaptain deleted the fix/14382-scope-read-rate-limit branch November 11, 2026 05:41

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This review checked head 5004346. The wait runs only in the two fail-closed reads that set the same-repo per-head floor. The floor and override decisions and the final exit 1 are unchanged. The Bedrock credential is minted after the override read, so the wait never holds a live credential. Both changed test classes pass at head. With the workflow hunk reverted, the two tests the Proof line names fail.

Done well: the tests run the real step scripts with date and sleep stubbed as shell functions. They pin an exact 105s wait and a fail-closed path past the budget, and the stubs also hold on Git for Windows bash.

Optional:

  • docs/ci/ci-and-reviews.md:4204 says a later reset "fails closed at once", but the 1s and 2s retries still run first, and your own test pins that. Suggested wording: "fails closed after the usual short retries, as before".
  • Tests do not fully pin the 1200s cap, its running total, or the raised timeout-minutes (50 and 40). Changing the cap to 3600, or putting the timeouts back to 30 and 20, keeps every test green. A boundary case (1200 waits, 1201 refused) and a timeout headroom check like the one in test_scheduled_gh_rate_limit.py would catch that.
  • No test runs the 60s secondary-limit branch. Also, wait_out_rate_limit is copied into two steps and nothing checks that the copies stay equal.

@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Nov 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants