Repository navigation
fix(ci): wait out a spent rate limit in security scope review reads - #33680
Conversation
UX Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing blocks: the fix stays inside the two reads whose failure becomes the head's floor, and the floor's decision is unchanged. Notes
[DESIGN-REVIEWED] 5004346 Description read[DESCRIPTION-READ] 9618bc86aef6582852951c35116ec60185779532b5ed4ee45b82f839f1d75e60 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS This fix aims at the cause: a read that hits a spent rate limit gives up before the reset and floors the head. One copy of the wait duplicates an existing repo mechanism, which is a note, not a block. Notes
What this change shipsInventory (7 items) — 6 justifiedIntent: stop a same-repo Security Scope Review head going permanently red because the shared API limit was spent when it ran. FIX.
[FIRST-PRINCIPLES-REVIEWED] 5004346 Description read[DESCRIPTION-READ] a42fc9eca415369b0bfde8002d8a5dcf81730b09818ea8c5547bb9fc609e3cfd That sha256 names the PR description this verdict read. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Review detailsNo findings. Notes:
[OPUS-REVIEWED] 5004346 Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
|
[operator: bolichen97] Bundle Size Gate is red here because of main, not this diff (it touches no frontend file): the App chunk is over its ceiling on main, tracked in #33668, with fixes open in #33662, #33665 and #33669. This PR waits for one of those to merge, then rebases onto it. The earlier red review lanes on this head were installation rate-limit casualties (reset 04:13:28Z) and were re-run once after the reset. |
On a same-repo pull request, Security Scope Review read the PR comments (for a human override) and the head's earlier check-runs (for the per-head floor) three times within three seconds. A spent installation rate limit resets within the hour, so all three tries failed, the run failed closed, and that failure became the head's floor: a later re-run that measured success was held at it. Both reads now wait for the limit's reset when it is within 20 minutes, the budget the shared gh-rate-limit action uses; a later reset still fails closed at once. What the floor decides does not change. The two jobs' timeouts grow by the wait. Refs #14382
06d13a9 to
5004346
Compare
|
Rebased: main moved in docs/ci/ci-and-reviews.md and in workflows beside .github/workflows/security-scope-review.yml since this head's CI ran (2caa69b -> 93ca1d9); CI re-running. The new base carries the Bundle Size Gate fix (#33665) this head was waiting on. The same push fixes this PR's two new rate-limit tests on the Windows shards: they now stub [operator: bolichen97] |
buluoray
left a comment
There was a problem hiding this comment.
Approve on 5004346. The fail-closed gate is not weakened: the change only adds a bounded wait before re-reading, a read that never succeeds still exits 1 and the floor's inputs and decision are unchanged. The Windows test failures that blocked the prior head (06d13a9) are fixed by turning the sleep/date PATH stubs into shell functions, and every Windows lane is green on this head. Decisive lines: .github/workflows/security-scope-review.yml:307-335 (override read: wait only on a "rate limit" stderr, 1200 s cap, read_ok still set only by a successful read, exit 1 after 3 failures unchanged), .github/workflows/security-scope-review.yml:3009-3037 (floor read: same wait, readable="false" path and floor decision at :3046-3060 unchanged), test/test_ai_review_workflows_human_override.py (sleep/date as shell functions), test/test_ai_review_workflows_scope_verdict.py (same)
Pull request was closed
|
[operator: bolichen97] Closed and reopened with no code change: PR Readiness was waiting on a Code Review run that records its base (the #33337 rule), and only a new pull request event starts one. The close turned auto-merge off; it is re-armed once this re-run is green. |
|
[operator: bolichen97] Re-ran CI on a fresh merge ref by closing and reopening this PR; nothing was pushed and the diff is unchanged. The last run's only reds were |
KiroCrewCaptain
left a comment
There was a problem hiding this comment.
This review checked head 5004346. The wait runs only in the two fail-closed reads that set the same-repo per-head floor. The floor and override decisions and the final exit 1 are unchanged. The Bedrock credential is minted after the override read, so the wait never holds a live credential. Both changed test classes pass at head. With the workflow hunk reverted, the two tests the Proof line names fail.
Done well: the tests run the real step scripts with date and sleep stubbed as shell functions. They pin an exact 105s wait and a fail-closed path past the budget, and the stubs also hold on Git for Windows bash.
Optional:
- docs/ci/ci-and-reviews.md:4204 says a later reset "fails closed at once", but the 1s and 2s retries still run first, and your own test pins that. Suggested wording: "fails closed after the usual short retries, as before".
- Tests do not fully pin the 1200s cap, its running total, or the raised
timeout-minutes(50 and 40). Changing the cap to 3600, or putting the timeouts back to 30 and 20, keeps every test green. A boundary case (1200 waits, 1201 refused) and a timeout headroom check like the one in test_scheduled_gh_rate_limit.py would catch that. - No test runs the 60s secondary-limit branch. Also,
wait_out_rate_limitis copied into two steps and nothing checks that the copies stay equal.
Problem / Motivation
Goal: A same-repo pull request whose Security Scope Review run meets a spent GitHub API rate limit that resets within 20 minutes gets a measured verdict after the reset, instead of a red head that no re-run can clear.
On a same-repo pull request, Security Scope Review reads the PR's comments (to find a human override) and the head's earlier check-runs (for the per-head floor). Each read tried three times within about three seconds. When the shared installation rate limit was spent, all three tries failed, the run failed closed, and that failure became the head's floor: a later re-run that measured
successwas held at it. This happened on #13647 (the comment read), #13335 and #32110's PR #30937 (the check-run read).Why it matters
Every same-repo author whose push lands while the shared token is drained gets a red Security Scope Review that a re-run cannot clear. Today the only ways out are a new push, which re-rolls every other lane and often meets the same limit, or a writer's
/ai-review override scope <sha>.Not a goal
/ai-review override scope <sha>, as does a head a past flake already floored.What changed (motivation → approach → change)
A same-repo pull request whose Security Scope Review run meets a drained API token is no longer left red for good when the limit resets within 20 minutes: the run waits for the reset and then measures the change. The PR-comment read and the earlier check-runs read used to give up after about three seconds, but a spent installation rate limit resets within the hour, so the run failed closed and the per-head floor kept that failure.
Both reads wait for the reset when a failed try's error names a rate limit. They read the REST limit's reset time from
gh api rate_limit(which costs no quota), sleep until 5 seconds past it (60 seconds when the REST limit is not spent, which means a secondary limit), and try again. Each read waits at most 20 minutes in all, the same budget.github/actions/gh-rate-limitand the fork Opus lane use; a reset further away than that fails closed exactly as before. Any other error keeps the old short retry. The wait is written inline in each step becausegenerateruns no repository program while it holds the Bedrock credential.The
generatejob's timeout goes from 30 to 50 minutes and thepublishjob's from 20 to 40, so the wait fits inside the job. Security Scope Review's fail-closed ruling indocs/ci/ci-and-reviews.mdsays the two reads wait, and that the wait never changes what the floor decides.Backwards compatibility
Compatible: a read that succeeds, or fails for any reason other than a rate limit, behaves exactly as before; a rate-limited read waits and then gives the same answer or the same fail-closed error.
Removes nothing: no check, output or verdict is dropped; the floor's inputs and its decision are unchanged.
Tests
test/test_ai_review_workflows_human_override.py::TestOverrideReadFailureFailsClosed::test_scope_lane_waits_out_a_spent_rate_limitruns the real override read withgh,dateandsleepstubbed: a rate-limited first try waits 105 s (reset 100 s away, plus 5) and then resolves the record; a reset past the 20-minute budget gets no long wait and still fails closed after three tries.test/test_ai_review_workflows_scope_verdict.py::TestPerHeadMonotonicFloor::test_same_repo_floor_read_waits_out_a_spent_rate_limitruns the real floor step: a rate-limited first listing read waits 105 s, then the floor settles on the listing and the step passes.Proof: PROVEN test/test_ai_review_workflows_human_override.py::TestOverrideReadFailureFailsClosed::test_scope_lane_waits_out_a_spent_rate_limit[1100-1-0-0-105\n-xx], test/test_ai_review_workflows_scope_verdict.py::TestPerHeadMonotonicFloor::test_same_repo_floor_read_waits_out_a_spent_rate_limit (head 5004346)
Manual verification
Both edited step scripts pass
bash -n. Not run against a live spent limit, which cannot be produced on demand; the tests above execute the two real step scripts withgh,dateandsleepstubbed.Screenshots / video
N/A: no user-visible surface changes; this is a CI workflow, its tests and one docs paragraph.
Related Issues
Refs #14382
Sibling sweep
Swept: git grep -nE "for attempt in" -- .github/workflows/security-scope-review.yml .github/workflows/fork-security-scope-review.yml
Sibling: .github/workflows/security-scope-review.yml:2449 -- not affected -- the verdict post's comment reads end their step with exit 0 and keep an unposted verdict as an artifact, so they never fail the run
Sibling: .github/workflows/fork-security-scope-review.yml:1850 -- not affected -- a fork run that cannot read its priors marks its own check-run unsettled and sets no per-head floor, and a failed fork override read (:316) warns and reviews normally
Pattern harvest
Not generalizable: the hazard is specific to a gate whose own failed run becomes its next input (the same-repo per-head floor); other lanes' failed runs are cleared by a re-run.
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)