Repository navigation
ci: hold pr readiness on the sensitive change review run - #36483
Conversation
UX Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing blocks: the lane is enrolled from the one run a PR cannot fake, and the publish re-check reads the same run. Notes
[DESIGN-REVIEWED] ec8fb63 Description read[DESCRIPTION-READ] d81551a9206493446ec8ed25755caa3bf2dfe704a32b96545bd90d117fa4deca That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Nothing blocks. Each item closes the gap the base spec describes ("enrolling it is the rest of #16870"), and each new read uses the existing What this change shipsInventory (7 items) — 7 justifiedIntent: stop a sensitive-path PR from reaching
[FIRST-PRINCIPLES-REVIEWED] ec8fb63 Description read[DESCRIPTION-READ] 4c2ec2a73cd4a2715222043632d4b1204c534bb7c396ca26b6b77b229a0c1a44 That sha256 names the PR description this verdict read. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findings (all downgraded on adjudication)GPT 6.1 flagged blocking issues on This comment is updated in place on each push. Review detailsBLOCKING -- .github/workflows/pr-readiness.yml:2371 -- A sibling PR’s sensitive review can replace this PR’s failure
Adjudication (Opus 5.5) — is blocking on each finding proportionate?F1 — LOW harm. The gate's own contract makes the harm low. The Sensitive Change Review check only verifies format, and the PR author's own comment satisfies it ( False positive or not applicable? A repository writer can comment: |
Code change approver recommendation: A person should decideWhy: the decider thinks the change is probably fine but estimates a 15% chance that this change introduces a defect, above the 12% limit for approving on its own. Readiness now enrolls Sensitive Change Review and reads it from the head's An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit |
408d76a to
0a391c7
Compare
|
Rebased: main moved in docs/ci/ci-and-reviews.md, test/test_pr_readiness_evaluate.py and 16 other files beside this PR's since this head was built (2f7e9f2 -> c22557c); CI re-running. The previous head's Backend Lint & Type Check red (mypy at |
|
self-added: no
|
|
Review passed on this head. Please rebase onto the latest main so CI can run fresh; this head is 606 commits behind main and its CI ran on 11-13. The approval will follow once the rebased head is green. |
PR Readiness, the one required status on main, never read Sensitive Change Review: its evaluate step enrolled only pull_request lanes, so a sensitive-path PR whose review record was missing (a red run) still published success and auto-merge could land it. Enroll the lane and read it from the head's pull_request_target runs of sensitive-change-review.yml, bound to the PR's head repository and branch. GitHub runs those from the default branch's file whatever the PR's base, so a check another workflow posts under the same name, or a pull_request run of a PR's own copy of the file, leaves the lane pending instead of passing it. The lane joins the edited-lanes list because its trigger hears `edited`. The publish step's pre-success re-check reads the same pull_request_target run, so a lane re-run that turns red between evaluation and publish publishes failure (or pending while it runs) instead of a stale success. The readiness test harness now answers the pull_request_target read from its own fixture, the transport-failure substring stops matching it, and the subscription-list test exempts this lane, which is not a workflow_run subscription by design. The awaiting-approval tests move to their own module to keep the evaluate tests under the file line cap. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Rebased: main moved in .github/workflows/pr-readiness.yml, docs/ci/ci-and-reviews.md and other files beside this PR's since this head was built (c22557c -> 433d99b); CI re-running. @buluoray thanks for the review: this is the rebase you asked for. The patch is unchanged (patch-id identical to 0a391c7), and every local gate passed again on the rebased tree. |
0a391c7 to
ec8fb63
Compare
KiroCrewCaptain
left a comment
There was a problem hiding this comment.
Approving. On this head the review checked four things. The new lane is read only from pull_request_target runs of sensitive-change-review.yml, bound by path, head repo and head branch. A same-named check or a pull_request run of the PR's own copy leaves the lane pending. The publish step re-reads the lane before it posts success. Mutation runs on a scratch copy confirmed the tests pin both production hunks: reverting the evaluate read turned 4 of the 5 new sensitive-review tests red, and emptying the publish re-read turned both parametrized publish cases red. Done well: the change reuses gh_head_runs, the cancelled-twin collapse and the existing transport-failure path, so it adds no new mechanism.
Security: this edits the readiness gate that enforces the Sensitive Change Review record, and it only tightens that gate. Main ignores the lane. This PR holds a PR whose run is red, running or unreadable. A failed read falls into the existing [read-failed] pending branch, never into a pass.
- Optional:
test/test_pr_readiness_publish.py:126adds afail_pt_runsstub branch that no test sets. A test that publishes with that flag and expectspendingplus[read-failed]would pin the fail-closed path in the publish re-check. - Optional: the body says the awaiting-approval tests moved unchanged. The moved helper also plants an in_progress
pull_request_targetrun. A short PR comment would make that clear. - Optional: the GPT finding about a sibling PR on the same head branch has no disposition reply on this head. It uses the same binding as every other lane, but a one-line disposition would close it.
Problem / Motivation
Goal: A sensitive-path PR cannot reach
readiness: passedwhile its Sensitive Change Review run is red.PR Readinessis the only required status onmain, and its evaluate step never read Sensitive Change Review. It enrolls lanes inworkflow_specsand reads only the head'spull_requestruns. Sensitive Change Review runs onpull_request_targetand is in neither. So a PR that touchessandbox.pywith no reasoned review record had a red Sensitive Change Review run and still gotPR Readiness: success/readiness: passed, and armed auto-merge could land it.Why it matters
The sensitive-path record exists so changes to the security ceiling are reviewed with a reason. While readiness ignores it, the gate is advisory only: a red run blocks nothing and merges look clean.
Not a goal
workflow_runsubscription to this workflow. Readiness already hears itspull_request_targetevents, and the sweep recomputes when a re-run completes.What changed (motivation → approach → change)
pull_request_target) was never read.pull_request_targetruns of.github/workflows/sensitive-change-review.yml, bound to the PR's head repository and branch with the same (path, head repo, head branch) collapse every other lane uses. Never read it as a check by name. GitHub runs those from the default branch's file whatever the PR's base (a release-branch PR included), so a check another workflow posts under that name, or apull_requestrun of a PR's own edited copy, leaves the lane pending instead of passing it. The read goes through the existinggh_head_runshelper (retry, empty-listing fallback through check-runs filtered by event + head SHA). A failed read follows the same transport-failure path as the other lanes (non-terminal "could not be evaluated"; HTTP 22 fails loud).pull_request_targetrun, bound and collapsed the same way. A re-run that turned red between evaluation and publish publishesfailure; one still running publishespending. Apull_requestrun of the PR's own copy is never read there either..github/workflows/pr-readiness.yml: oneworkflow_specsentry; one extra read for that lane in the run loop and one in the publish re-check;sensitive-change-review.ymladded toedited_lanes, because its trigger listsedited.docs/ci/ci-and-reviews.md: the lane is described as blocking, read from itspull_request_targetrun, and re-read before a success is published; it is added to the always-required list and the aggregate list.ghstubs answer thepull_request_targetread from their own fixture (default: one green run). The transport-failure substring is pinned toevent=pull_request&so it no longer also matches the new read. The empty-listing fallback test gives this lane apull_request_targetrun. The workflow_run subscription-list test exempts this lane, whichpr-readiness.ymlalready documents as deliberately not subscribed. The awaiting-approval tests move unchanged totest/test_pr_readiness_awaiting_approval.py, to keeptest_pr_readiness_evaluate.pyunder the 2,000-line cap.Backwards compatibility
Breaking: readiness now holds a PR whose Sensitive Change Review run is red (before, the gate was advisory). A head last pushed before the gate moved to
pull_request_targethas no such run and readsSensitive Change Review (not started on this head; push or reopen)until a push or reopen. Writer sweep on origin/main @ 433d99b: the only writer of that run issensitive-change-review.ymlitself, re-run bysensitive-change-review-comment.yml/sensitive-change-review-approval.yml. The readers of the readiness verdict (pr-readiness-sweep.yml,pr_status.py, auto-merge) are unchanged.Reader: .github/workflows/pr-readiness-sweep.yml:sweep -- release -- re-fires the recompute when a check on the head completes, including this run; test_pr_readiness_sweep.py
Reader: src/kiro_crew/builtin_skills/kirocrew-dev/kirocrew-prepare-pr/scripts/pr_status.py:resolve_readiness_context -- chat -- reads the
PR Readinessstate only, whose shape is unchanged; test_prepare_pr_status.pyTests
Proof: PROVEN test/test_pr_readiness_publish.py::test_a_sensitive_review_re_run_since_the_evaluation_holds_the_success[completed-failure-failure-readiness: action required], test/test_pr_readiness_publish.py::test_a_sensitive_review_re_run_since_the_evaluation_holds_the_success[in_progress-success-pending-readiness: checking], test/test_pr_readiness_sensitive_review.py::test_the_pull_request_target_run_decides_the_lane[success], test/test_pr_readiness_sensitive_review.py::test_the_pull_request_target_run_decides_the_lane[failure], test/test_pr_readiness_sensitive_review.py::test_a_same_named_check_from_another_run_does_not_pass_the_lane (head 0a391c7)
test/test_pr_readiness_sensitive_review.py(new): thepull_request_targetrun decides the lane (success passes it; failure reds readiness); a same-named green check plus a greenpull_requestrun of the PR's own copy leave the lane pending; a deadpull_request_targetread is the non-terminal "could not be evaluated", never a pass; a PR to a release branch is read the same way.test/test_pr_readiness_publish.py::test_a_sensitive_review_re_run_since_the_evaluation_holds_the_success: with everypull_requestlane green, a redpull_request_targetrun publishesfailure/readiness: action requiredand a running one publishespending/readiness: checking, even with a greenpull_requestrun of the same path on the page.test_pr_readiness_publish.pyin 3 shuffled node-id orders green;test_flake_pattern_ratchet.pygreen. Every profile gate, semgrep andscripts/local-gate.pyexit 0.python -m pytest -n 4 test/test_pr_readiness_*.py test/test_ai_review_workflows*.py test/test_readiness_sweep_scan.py test/test_ci_*.py test/test_slot_ownership_checkpoint.py: 4340 passed.Manual verification
Reproduced and smoke-tested on Linux (Ubuntu). The new tests fail on the workflow without each change and pass on this head. The real "Evaluate current revision" and "Publish status and label" step texts, extracted the way the test harnesses do, were then driven against live public data through a read-only anonymous
ghshim (GET only, no token; every write recorded, none sent).Evaluate step (unchanged on this head):
pull_request_targetrunmainfailurefailed=[... Sensitive Change Review (failure) ...],status_state=failuremainsuccesspassed=[... Sensitive Change Review ...]release/0.11.0success(run 42626510449, from the default branch's file)passed=[... Sensitive Change Review ...]Publish step,
STATUS_STATE=success:pull_request_targetrunfailure(run 43405078700)failure:blocking readiness item(s) since evaluation: Sensitive Change Reviewpending, held only by a running CodeQL scansuccesssuccesssuccessRelated Issues
Closes #16870
Sibling sweep
Swept: git grep -n 'workflow_specs+=' -- .github/workflows/pr-readiness.yml; git grep -lE '^\s+pull_request_target:' -- .github/workflows/; git grep -nE 'gh_head_runs "$REPO" "$SHA"' -- .github/workflows/
Siblings: none -- Sensitive Change Review is the only readiness lane whose run is pull_request_target, and its two reads (evaluate, publish re-check) are both in this diff; every other enrolled lane runs on pull_request or is the managed CodeQL dynamic run, which the publish re-check already re-reads
Pattern harvest
Not generalizable: the class (a blocking lane missing from the readiness aggregate or its pre-publish re-check) is already pinned by the subscription-list and lane-file tests for
pull_requestlanes; this lane escaped them only because it runs on a different event, which is now explicit in the spec and both tests.Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)Contribution License Agreement
Contributions are accepted under the project's Apache-2.0 license; see the Licensing section of CONTRIBUTING.md.