Skip to content

ci: hold pr readiness on the sensitive change review run - #36483

Merged
bolichen97 merged 1 commit into
mainfrom
fix/16870-readiness-sensitive-review
Nov 17, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
fix/16870-readiness-sensitive-review

Conversation

@bolichen97

@bolichen97 bolichen97 commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A sensitive-path PR cannot reach readiness: passed while its Sensitive Change Review run is red.

PR Readiness is the only required status on main, and its evaluate step never read Sensitive Change Review. It enrolls lanes in workflow_specs and reads only the head's pull_request runs. Sensitive Change Review runs on pull_request_target and is in neither. So a PR that touches sandbox.py with no reasoned review record had a red Sensitive Change Review run and still got PR Readiness: success / readiness: passed, and armed auto-merge could land it.

Why it matters

The sensitive-path record exists so changes to the security ceiling are reviewed with a reason. While readiness ignores it, the gate is advisory only: a red run blocks nothing and merges look clean.

Not a goal

  • Changing what Sensitive Change Review checks, or which paths count as sensitive.
  • A workflow_run subscription to this workflow. Readiness already hears its pull_request_target events, and the sweep recomputes when a re-run completes.

What changed (motivation → approach → change)

  • Symptom → cause: a red Sensitive Change Review run left readiness green, because the lane was never enrolled and its run's event (pull_request_target) was never read.
  • Approach: enroll the lane, but read it from the head's pull_request_target runs of .github/workflows/sensitive-change-review.yml, bound to the PR's head repository and branch with the same (path, head repo, head branch) collapse every other lane uses. Never read it as a check by name. GitHub runs those from the default branch's file whatever the PR's base (a release-branch PR included), so a check another workflow posts under that name, or a pull_request run of a PR's own edited copy, leaves the lane pending instead of passing it. The read goes through the existing gh_head_runs helper (retry, empty-listing fallback through check-runs filtered by event + head SHA). A failed read follows the same transport-failure path as the other lanes (non-terminal "could not be evaluated"; HTTP 22 fails loud).
  • Before publishing a success, the publish step's re-check reads that same pull_request_target run, bound and collapsed the same way. A re-run that turned red between evaluation and publish publishes failure; one still running publishes pending. A pull_request run of the PR's own copy is never read there either.
  • .github/workflows/pr-readiness.yml: one workflow_specs entry; one extra read for that lane in the run loop and one in the publish re-check; sensitive-change-review.yml added to edited_lanes, because its trigger lists edited.
  • docs/ci/ci-and-reviews.md: the lane is described as blocking, read from its pull_request_target run, and re-read before a success is published; it is added to the always-required list and the aggregate list.
  • Tests: the evaluate and publish harnesses' gh stubs answer the pull_request_target read from their own fixture (default: one green run). The transport-failure substring is pinned to event=pull_request& so it no longer also matches the new read. The empty-listing fallback test gives this lane a pull_request_target run. The workflow_run subscription-list test exempts this lane, which pr-readiness.yml already documents as deliberately not subscribed. The awaiting-approval tests move unchanged to test/test_pr_readiness_awaiting_approval.py, to keep test_pr_readiness_evaluate.py under the 2,000-line cap.

Backwards compatibility

Breaking: readiness now holds a PR whose Sensitive Change Review run is red (before, the gate was advisory). A head last pushed before the gate moved to pull_request_target has no such run and reads Sensitive Change Review (not started on this head; push or reopen) until a push or reopen. Writer sweep on origin/main @ 433d99b: the only writer of that run is sensitive-change-review.yml itself, re-run by sensitive-change-review-comment.yml / sensitive-change-review-approval.yml. The readers of the readiness verdict (pr-readiness-sweep.yml, pr_status.py, auto-merge) are unchanged.

Reader: .github/workflows/pr-readiness-sweep.yml:sweep -- release -- re-fires the recompute when a check on the head completes, including this run; test_pr_readiness_sweep.py
Reader: src/kiro_crew/builtin_skills/kirocrew-dev/kirocrew-prepare-pr/scripts/pr_status.py:resolve_readiness_context -- chat -- reads the PR Readiness state only, whose shape is unchanged; test_prepare_pr_status.py

Tests

Proof: PROVEN test/test_pr_readiness_publish.py::test_a_sensitive_review_re_run_since_the_evaluation_holds_the_success[completed-failure-failure-readiness: action required], test/test_pr_readiness_publish.py::test_a_sensitive_review_re_run_since_the_evaluation_holds_the_success[in_progress-success-pending-readiness: checking], test/test_pr_readiness_sensitive_review.py::test_the_pull_request_target_run_decides_the_lane[success], test/test_pr_readiness_sensitive_review.py::test_the_pull_request_target_run_decides_the_lane[failure], test/test_pr_readiness_sensitive_review.py::test_a_same_named_check_from_another_run_does_not_pass_the_lane (head 0a391c7)

  • test/test_pr_readiness_sensitive_review.py (new): the pull_request_target run decides the lane (success passes it; failure reds readiness); a same-named green check plus a green pull_request run of the PR's own copy leave the lane pending; a dead pull_request_target read is the non-terminal "could not be evaluated", never a pass; a PR to a release branch is read the same way.
  • test/test_pr_readiness_publish.py::test_a_sensitive_review_re_run_since_the_evaluation_holds_the_success: with every pull_request lane green, a red pull_request_target run publishes failure / readiness: action required and a running one publishes pending / readiness: checking, even with a green pull_request run of the same path on the page.
  • Determinism: the new publish test 20 consecutive runs green, plus test_pr_readiness_publish.py in 3 shuffled node-id orders green; test_flake_pattern_ratchet.py green. Every profile gate, semgrep and scripts/local-gate.py exit 0.
  • python -m pytest -n 4 test/test_pr_readiness_*.py test/test_ai_review_workflows*.py test/test_readiness_sweep_scan.py test/test_ci_*.py test/test_slot_ownership_checkpoint.py: 4340 passed.

Manual verification

Reproduced and smoke-tested on Linux (Ubuntu). The new tests fail on the workflow without each change and pass on this head. The real "Evaluate current revision" and "Publish status and label" step texts, extracted the way the test harnesses do, were then driven against live public data through a read-only anonymous gh shim (GET only, no token; every write recorded, none sent).

Evaluate step (unchanged on this head):

PR Base Newest pull_request_target run Lane on this head Lane without the change
#36271 main failure failed=[... Sensitive Change Review (failure) ...], status_state=failure absent from every array
#36405 main success passed=[... Sensitive Change Review ...] absent
#33881 release/0.11.0 success (run 42626510449, from the default branch's file) passed=[... Sensitive Change Review ...] absent

Publish step, STATUS_STATE=success:

PR Newest pull_request_target run This head publishes Without the re-check
#36436 (monitored lanes narrowed to the ones green on that head) failure (run 43405078700) failure: blocking readiness item(s) since evaluation: Sensitive Change Review pending, held only by a running CodeQL scan
#36359 success success success

Related Issues

Closes #16870

Sibling sweep

Swept: git grep -n 'workflow_specs+=' -- .github/workflows/pr-readiness.yml; git grep -lE '^\s+pull_request_target:' -- .github/workflows/; git grep -nE 'gh_head_runs "$REPO" "$SHA"' -- .github/workflows/
Siblings: none -- Sensitive Change Review is the only readiness lane whose run is pull_request_target, and its two reads (evaluate, publish re-check) are both in this diff; every other enrolled lane runs on pull_request or is the managed CodeQL dynamic run, which the publish re-check already re-reads

Pattern harvest

Not generalizable: the class (a blocking lane missing from the readiness aggregate or its pre-publish re-check) is already pinned by the subscription-list and lane-file tests for pull_request lanes; this lane escaped them only because it runs on a different event, which is now explicit in the spec and both tests.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributions are accepted under the project's Apache-2.0 license; see the Licensing section of CONTRIBUTING.md.

@bolichen97
bolichen97 requested review from a team and KiroCrewCaptain as code owners November 13, 2026 07:40
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 13, 2026
@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision ec8fb637c182109d65f97c04a5029c654216179e touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of ec8fb637c182109d65f97c04a5029c654216179e — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: the lane is enrolled from the one run a PR cannot fake, and the publish re-check reads the same run.

Notes

  • Sensitive Change Review cancels its in-progress run on every edited event, so title or body edits can leave the newest run cancelled. The existing collapse (a cancelled newest run yields to the latest one that started) should cover this. It is worth a test, because an edit storm could leave only cancelled runs, and the lane would then stay pending.

[DESIGN-REVIEWED] ec8fb63

Description read

[DESCRIPTION-READ] d81551a9206493446ec8ed25755caa3bf2dfe704a32b96545bd90d117fa4deca

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of ec8fb637c182109d65f97c04a5029c654216179e — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Nothing blocks. Each item closes the gap the base spec describes ("enrolling it is the rest of #16870"), and each new read uses the existing gh_head_runs helper and the existing collapse logic rather than writing them again.

What this change ships

Inventory (7 items) — 7 justified

Intent: stop a sensitive-path PR from reaching readiness: passed while its Sensitive Change Review run is red — a FIX (#16870).

  1. A red Sensitive Change Review run now turns PR Readiness red and holds auto-merge — justified
  2. Before publishing a success, readiness re-reads that run, so a re-run that turned red or is still running holds the publish — justified
  3. The lane trusts only the pull_request_target run from the default branch's file; a check with the same name or a PR's own edited copy leaves it pending — justified
  4. A head last pushed before the gate moved shows "not started; push or reopen" — justified
  5. Editing a PR's description re-evaluates this lane (edited_lanes) — justified
  6. The CI doc now calls the lane blocking and adds it to the required and aggregate lists — justified
  7. The awaiting-approval tests move unchanged into a new file to stay under the line cap — justified

[FIRST-PRINCIPLES-REVIEWED] ec8fb63

Description read

[DESCRIPTION-READ] 4c2ec2a73cd4a2715222043632d4b1204c534bb7c396ca26b6b77b229a0c1a44

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed ec8fb637c182109d65f97c04a5029c654216179e — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] ec8fb63

Verdict parsed from the review's SHA-scoped output markers for commit ec8fb637c182109d65f97c04a5029c654216179e.

False positive or not applicable? A repository writer can comment:
/ai-review override fable ec8fb637c182109d65f97c04a5029c654216179e false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings (all downgraded on adjudication)

GPT 6.1 flagged blocking issues on ec8fb637c182109d65f97c04a5029c654216179e; Opus 5.5 adjudication downgraded every one of them. Downgraded: the remedy each one needs is disproportionate to the harm it prevents. Adjudication downgraded all 1 blocking finding(s): the remedy each one requires is disproportionate to the harm it prevents.

This comment is updated in place on each push.

Review details

BLOCKING -- .github/workflows/pr-readiness.yml:2371 -- A sibling PR’s sensitive review can replace this PR’s failure

| select(.path == ".github/workflows/sensitive-change-review.yml"))
Trigger: Two open PRs target different bases from the same repository, branch and SHA. This PR has no reasoning record; the sibling has one. Editing the sibling’s title creates a newer successful sensitive-review run without restarting Code Review. A readiness recompute for this PR selects that sibling run during evaluation and publish.
Outcome: With the remaining lanes green, this PR publishes success and readiness: passed despite its own Sensitive Change Review remaining red. The checker reads reasoning records separately from each PR’s comments and reviews.
Anchor: defect
Fix: Bind sensitive-review runs to the current PR using trusted PR provenance before collapsing them in both evaluation and publish; never substitute a sibling PR’s verdict.

[BLOCK-MERGE-DOWNGRADED] ec8fb63
[GPT-REVIEWED] ec8fb63

Adjudication (Opus 5.5) — is blocking on each finding proportionate?

F1 — LOW harm. The gate's own contract makes the harm low. The Sensitive Change Review check only verifies format, and the PR author's own comment satisfies it (.github/workflows/sensitive-change-review.yml:3-12: "It cannot tell a true reason from a false one"). So reaching it through a sibling PR gives no ability the author lacks, and the precondition is a writer-created same-repository branch with two PRs open from the same SHA. Conditions confirmed: runs are bound only by head repo and branch (pr-readiness.yml:1676-1680, 2372-2373); the sensitive lane's concurrency group is per PR number, so a sibling's run survives (sensitive-change-review.yml:35-36); the edited trigger exists (:29). Recovery: none automatic, but the PR's own Sensitive Change Review check stays visibly red.
The real fix is a separate PR-provenance binding just for this lane, in both evaluation (:1645-1682) and the publish re-check (:2366-2394). It would split away from the shared head binding every other lane uses, which is more than this low harm justifies.

[ADJUDICATION] ec8fb637c182109d65f97c04a5029c654216179e total=1 uphold=0 downgrade=1
DOWNGRADE F1 .github/workflows/pr-readiness.yml:2371 reason=disproportionate-remedy
[GPT-ADJUDICATED] ec8fb637c182109d65f97c04a5029c654216179e

False positive or not applicable? A repository writer can comment:
/ai-review override gpt ec8fb637c182109d65f97c04a5029c654216179e false-positive: <reason, 20 to 500 characters>

@bolichen97
bolichen97 enabled auto-merge (squash) November 13, 2026 07:50
@KiroCrewCaptain

KiroCrewCaptain commented Nov 13, 2026 •

Copy link
Copy Markdown
Collaborator

Code change approver recommendation: A person should decide

Why: the decider thinks the change is probably fine but estimates a 15% chance that this change introduces a defect, above the 12% limit for approving on its own.

Readiness now enrolls Sensitive Change Review and reads it from the head's pull_request_target runs of the default-branch file. Those runs are filtered to the workflow's path, the PR's head repository and head branch, and reduced to the newest run the same way every other lane is. They go through the existing gh_head_runs helper, and a failed read is handled like any other lane's. The publish step's pre-success re-check now also reads those runs, filtered by path only to sensitive-change-review.yml, so a red or running re-run downgrades the success. This fixes the GPT lane's earlier blocking finding. The edited_lanes addition matches the workflow's trigger. The sweep picks up re-run completions from the head's check runs, and the re-check's pass/fail rules match the evaluation's. Docs match the code. The tests cover the green and red runs, a same-named check or own-copy run not passing the lane, an unreadable read, release-branch PRs and the publish re-check. I found nothing that blocks.

An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit 0a391c7365 · 476 changed lines. If this recommendation is wrong, reply here: replies are how the approver is evaluated.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 13, 2026
@bolichen97
bolichen97 force-pushed the fix/16870-readiness-sensitive-review branch from 408d76a to 0a391c7 Compare November 13, 2026 11:10
@bolichen97

Copy link
Copy Markdown
Contributor Author

Rebased: main moved in docs/ci/ci-and-reviews.md, test/test_pr_readiness_evaluate.py and 16 other files beside this PR's since this head was built (2f7e9f2 -> c22557c); CI re-running.

The previous head's Backend Lint & Type Check red (mypy at src/kiro_crew/cli_commands_memory.py:335), and the Coverage Gate that fails closed on it, were main's and are fixed on main by #36457; this head carries that fix. The CodeQL Analyze (python) red was an upload rate limit, not a finding. The same push answers the GPT 6.1 finding (disposition below).
[operator: bolichen97]

@bolichen97

Copy link
Copy Markdown
Contributor Author

self-added: no
mechanism: the publish step's pre-success re-check re-reads the sensitive lane from its pull_request_target run

  • Recheck the sensitive lane before publishing success span=0a5695a234e3

fixed in 0a391c7: the publish step's success re-check makes one more gh_head_runs "$REPO" "$SHA" pull_request_target "$runs_page" read and keeps only .github/workflows/sensitive-change-review.yml runs from it.
Those runs are bound to HEAD_REPO/HEAD_REF and collapsed the way the evaluation collapses them (max id; a cancelled max-id run yields to a later-started sibling), then folded into the same red/open result: red publishes failure, running publishes pending, a failed read takes the existing [read-failed] pending.
A pull_request run of that path (a PR's own copy) is never read by the re-check: MONITORED_LANES does not list it. runs_page is unchanged for the fork and CodeQL arms.
Pinned by test_pr_readiness_publish.py::test_a_sensitive_review_re_run_since_the_evaluation_holds_the_success (red -> failure, running -> pending, beside a green pull_request decoy of the same path); both cases fail on the previous workflow.
[operator: bolichen97]

@buluoray

Copy link
Copy Markdown
Contributor

Review passed on this head. Please rebase onto the latest main so CI can run fresh; this head is 606 commits behind main and its CI ran on 11-13. The approval will follow once the rebased head is green.

PR Readiness, the one required status on main, never read Sensitive
Change Review: its evaluate step enrolled only pull_request lanes, so a
sensitive-path PR whose review record was missing (a red run) still
published success and auto-merge could land it.

Enroll the lane and read it from the head's pull_request_target runs of
sensitive-change-review.yml, bound to the PR's head repository and
branch. GitHub runs those from the default branch's file whatever the
PR's base, so a check another workflow posts under the same name, or a
pull_request run of a PR's own copy of the file, leaves the lane pending
instead of passing it. The lane joins the edited-lanes list because its
trigger hears `edited`.

The publish step's pre-success re-check reads the same
pull_request_target run, so a lane re-run that turns red between
evaluation and publish publishes failure (or pending while it runs)
instead of a stale success.

The readiness test harness now answers the pull_request_target read
from its own fixture, the transport-failure substring stops matching
it, and the subscription-list test exempts this lane, which is not a
workflow_run subscription by design. The awaiting-approval tests move to
their own module to keep the evaluate tests under the file line cap.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@bolichen97

Copy link
Copy Markdown
Contributor Author

Rebased: main moved in .github/workflows/pr-readiness.yml, docs/ci/ci-and-reviews.md and other files beside this PR's since this head was built (c22557c -> 433d99b); CI re-running.

@buluoray thanks for the review: this is the rebase you asked for. The patch is unchanged (patch-id identical to 0a391c7), and every local gate passed again on the rebased tree.
[operator: bolichen97]

@bolichen97
bolichen97 force-pushed the fix/16870-readiness-sensitive-review branch from 0a391c7 to ec8fb63 Compare November 17, 2026 11:01
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Nov 17, 2026

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving. On this head the review checked four things. The new lane is read only from pull_request_target runs of sensitive-change-review.yml, bound by path, head repo and head branch. A same-named check or a pull_request run of the PR's own copy leaves the lane pending. The publish step re-reads the lane before it posts success. Mutation runs on a scratch copy confirmed the tests pin both production hunks: reverting the evaluate read turned 4 of the 5 new sensitive-review tests red, and emptying the publish re-read turned both parametrized publish cases red. Done well: the change reuses gh_head_runs, the cancelled-twin collapse and the existing transport-failure path, so it adds no new mechanism.

Security: this edits the readiness gate that enforces the Sensitive Change Review record, and it only tightens that gate. Main ignores the lane. This PR holds a PR whose run is red, running or unreadable. A failed read falls into the existing [read-failed] pending branch, never into a pass.

  • Optional: test/test_pr_readiness_publish.py:126 adds a fail_pt_runs stub branch that no test sets. A test that publishes with that flag and expects pending plus [read-failed] would pin the fail-closed path in the publish re-check.
  • Optional: the body says the awaiting-approval tests moved unchanged. The moved helper also plants an in_progress pull_request_target run. A short PR comment would make that clear.
  • Optional: the GPT finding about a sibling PR on the same head branch has no disposition reply on this head. It uses the same binding as every other lane, but a one-line disposition would close it.

@bolichen97
bolichen97 merged commit 6dd5dac into main Nov 17, 2026
100 checks passed
@bolichen97
bolichen97 deleted the fix/16870-readiness-sensitive-review branch November 17, 2026 12:58
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Nov 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ci: run Sensitive Change Review from the default branch

3 participants