Skip to content

fix(ci): name the scope-review validate refusal on the check-run itself - #13647

Merged
iamwhatever merged 1 commit into
mainfrom
fix/security-review-refusal-cause-13617
Sep 30, 2026
Merged

iamwhatever merged 1 commit into
mainfrom
fix/security-review-refusal-cause-13617

Conversation

@chenmingwei23

@chenmingwei23 chenmingwei23 commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

When the security-scope review refuses to run because a candidate operation carries a credential shape (rc=corpus-credential) or the normalized corpus could not be probed (rc=corpus-uncheckable), no adjudication leg is dispatched, so the fold reads no-report. The lane's generic sentence for that state — "no leg reported, and this was not the nothing-to-adjudicate short circuit" — is true of a broken lane, of a regression in the author's own diff, and of these refusals alike. The real cause reached an author only through a job annotation, four hops from the check they read.

Goal: Name a validate refusal cause on the Security Scope Review check itself, so an author reads why no leg ran without leaving the check.

Why it matters

A blocking check whose failure summary names no actionable cause sends the author to the wrong place: they debug the lane or their own diff while the fix is to re-propose the operation without the embedded credential shape. The cause and its remedy now travel to the shared conclusion table, so both the fork and same-repo lanes surface the same explanation and the same remedy in one place instead of a per-lane annotation.

Not a goal

The refusal itself is not relaxed by naming it. The corpus travels to the adjudication legs as a public artifact and must stay byte-faithful for the classifier, so a credential shape in it is still neither published nor rewritten. Naming the cause never softens the verdict — an unmeasured run fails closed exactly as it did while it was unexplained.

What changed (motivation → approach → change)

  • Symptom: a validate refusal produced a generic "no leg reported" summary on the Security Scope Review check.
  • Root cause: the refusal code (rc=corpus-*) was written as a job annotation but never carried into the shared conclusion table, so the table could not name the cause on the check-run.
  • Change: scope_candidates.py gains a _VALIDATE_REFUSALS table mapping each rc=corpus-* code to a short cause (for the 255-char check-run title) and a full remedy (for the summary). conclude_lane selects the cause by exact key match on a run whose fold is no-report; an unrecognized code falls through to the generic sentence. The fork lane passes the code through conclude --refusal and prints a remedy= line the publish step reads into the check-run summary.

Tests

  • test_scope_candidates.py: the named refusals surface their own cause and remedy, an unnamed code falls through to the generic sentence, naming a cause never softens the verdict, and a refusal outranks the green nothing-new row.
  • test_ai_review_workflows.py: the fork lane hands the refusal code to the table and interpolates no cause text in shell; and the newly added coupling test pins the rc=corpus-* codes the fork lane emits to the exact keys of _VALIDATE_REFUSALS, so a rename on either side fails a test rather than silently regressing to the generic sentence.

Manual verification

N/A — unit coverage sufficient. The first-principles reviewer's advisory note (verify on a real fork run that "this run's logs are linked from this check") concerns the check-run's details_url, which the publish step stamps with the run URL; it is a PASS-verdict observation, not a requested change.

Pattern harvest

Rule candidate: review-prompt
Pattern: a stringly-typed code emitted by a workflow and consumed by a lookup table in a script has no test coupling the two, so a rename on either side degrades silently to a fallback. Pin the emitted set against the table's keys, both directions.

Related Issues

Closes #13617

@chenmingwei23
chenmingwei23 requested review from a team and krishdhasmana September 25, 2026 02:26
@github-actions

github-actions Bot commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ✅ no blocking findings

GPT 5.6 completed its review of 232a0ba126fd81c77c4a9267598287dfb35fcff7 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 232a0ba

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 232a0ba126fd81c77c4a9267598287dfb35fcff7: <one-sentence reason>

@github-actions github-actions Bot added the readiness: action required A blocking check or review needs attention label Sep 25, 2026
@github-actions

github-actions Bot commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — 🟡 CONCERNS

Design-level review of 232a0ba126fd81c77c4a9267598287dfb35fcff7 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: CONCERNS

Same-repo lane still shows the generic sentence: the description claims both lanes surface the refusal cause, but only the fork lane is wired.

Watch

  • Description → diff gap. "Why it matters" says "both the fork and same-repo lanes surface the same explanation and the same remedy in one place instead of a per-lane annotation," but security-scope-review.yml is untouched: its validate step hits the identical refusals (probe=10 credential shape, unprobeable corpus) via exit 1 + ::error:: and its conclude --lane same-repo call (security-scope-review.yml:1751) passes no --refusal, so a same-repo author on a validate refusal still lands on the generic "no leg reported" check-run — the exact harm this PR names. The same-repo ::error:: prose is also a near-verbatim second copy of the corpus-credential remedy that the new coupling test does not pin, so the two texts can drift.
    Clears when: the same-repo lane emits the rc=corpus-* code and passes --refusal to its conclude call (retiring the duplicated YAML prose), or the description's both-lanes claim is corrected to fork-only with the same-repo wiring filed as a follow-up.

[DESIGN-REVIEWED] 232a0ba

Description read

[DESCRIPTION-READ] b171a28808c0af0b87a9abc5251dd36a7ab8ab73911f1f4269e398cd051cc51c

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5) — 🟡 CONCERNS

Premise-level review of 232a0ba126fd81c77c4a9267598287dfb35fcff7 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: CONCERNS

Description says both lanes surface the refusal's cause; only the fork lane is wired — the same-repo lane's identical refusals still read the generic sentence.

Not justified as shipped

  • Item 1 — symptom-level: point patch. "both the fork and same-repo lanes surface the same explanation and the same remedy in one place" — but grepping rc=corpus- across .github/workflows matches only the fork file: the same-repo lane's two identical refusal branches (security-scope-review.yml:1048, :1058) emit no code, and its conclude --lane same-repo call (:1751) passes no --refusal, so a same-repo refusal still folds to the generic "no leg reported" sentence via a job annotation — the exact defect ci: fork Security Scope Review reports 'no leg reported' without the validate refusal cause #13617 names. 2 sibling branches, 0 wired.
    Clears when: the same-repo refusal branches hand their code to the shared table, or a maintainer records the same-repo half as deliberately deferred.
  • Item 4 — undeclared, rides along: the generic no-rows summary for every unfolded run is reworded (drops the pointer to a PR comment that is never posted on this arm); harm-removing, but the description never mentions it.

What this change ships

Inventory (6 items) — 4 justified

Intent: FIX — make the Security Scope Review check itself say why a validate refusal ran no leg (Closes #13617).

  1. Fork check-run title names a refusal's cause instead of the generic sentence — symptom-level (2 same-repo siblings unwired)
  2. Fork check-run summary carries the refusal's full remedy text — justified
  3. conclude gains optional --refusal; unnamed codes fall through unchanged — justified
  4. Generic no-rows summary reworded, no longer points at an unposted PR comment — undeclared, rides along
  5. New helper validate_refusal_remedy (one consumer, _cmd_conclude) — justified
  6. A refusal now outranks the green nothing-new row in the table — justified

[FIRST-PRINCIPLES-REVIEWED] 232a0ba

Description read

[DESCRIPTION-READ] adc77403cf1e0a965a535bb59c051a2977f81d0595696aa901d2a174c1408f52

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 25, 2026
@github-actions

github-actions Bot commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5 Review — ✅ no blocking findings

Reviewed 232a0ba126fd81c77c4a9267598287dfb35fcff7 — this comment is updated in place on each push.

Review details

No findings that block.

FINDING — .github/workflows/fork-security-scope-review.yml:2471 — the unconditional "No leg produced confirmed rows, so there are none to read" is published on the fold=error path too, where verdict raised on a missing required leg (windows leg dies → rc=2 → fold="error" → title "the per-platform reports could not be folded", $BODY never written, REMEDY empty), so the author is told no leg confirmed a row while the ubuntu/macos reports retained as artifacts may each name a newly-refused operation → Fix: gate the second sentence on $FOLD_RC (already in this step's env) — keep the "no rows exist" claim for skipped/no-report, and on error say the per-platform reports could not be folded so any rows they carry were not published.

[OPUS-REVIEWED] 232a0ba

Verdict parsed from the review's SHA-scoped output markers for commit 232a0ba126fd81c77c4a9267598287dfb35fcff7.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 232a0ba126fd81c77c4a9267598287dfb35fcff7: <one-sentence reason>

@github-actions

github-actions Bot commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

Security Scope Review -- 🟡 CONCERNS

Which legitimate operations does 232a0ba126fd81c77c4a9267598287dfb35fcff7 newly refuse? Updated in place on each push. A script-confirmed regression blocks; the model proposes candidates and never decides a refusal.

Verdict basis: the reviewer's own judgement; no confirmed regression.

Security scope review -- adjudicated candidates

  • macos-latest (platform posix): 8 classified, 0 newly refused, 8 still allowed.
  • ubuntu-latest (platform posix): 8 classified, 0 newly refused, 8 still allowed.
  • windows-latest (platform windows): 2 classified, 0 newly refused, 2 still allowed.

No candidate flipped from allowed to refused.

Scope-Verdict: CONCERNS

TIGHTENING: the shared conclusion ladder in scripts/scope_candidates.py::conclude_lane — a named validate refusal row is inserted ahead of the green nothing-new rows ("Were they ever to arrive together, the refusal is the half that must win: the green nothing-new reading would publish a pass for a corpus no leg ever saw") — the ONE threat it refuses is a false-green check-run published over a corpus no adjudication leg ever measured. The underlying refusals themselves (rc=corpus-credential, rc=corpus-uncheckable) exist unchanged in the base workflow; this diff only names their cause and remedy in the published verdict. The tool-call deny composite (path fence, sensitive tier, exfil shapes, rule catalog) is not touched by any hunk.

ADJUDICATED: 10 candidates, 0 confirmed newly-refused
No confirmed rows were handed back for this run, and a regression without a verdict is not a finding. The candidates probe the boundary anyway: the new conclude --refusal invocation both lanes now run, the new remedy= readback and $GITHUB_OUTPUT append the decide step executes on every run, the read-only greps a maintainer runs over the new credential-vocabulary strings, and the test file the change extends.

PLATFORM COVERAGE: posix — the workflow's own additions (python3 … conclude --refusal, the printf | sed -n 's/^remedy=//p' readback, the $GITHUB_OUTPUT append) run in bash on the lane's runner, and each got a posix row in its exact spelling. windows — the diff's Python-side table is host-neutral but maintainers reproduce the decide step and run the extended test file on Windows, so the py -3 + backslash CLI spelling and the .venv\Scripts\python.exe pytest spelling each got a row. No any rows were written, because every candidate carries a path, separator, or executable name. I see no platform spelling the diff assumes that the candidates leave unprobed; the per-leg verdicts are the deterministic fold's to report, not mine.

UNADJUDICATED: the tightening lives entirely in the lane's own verdict policy, which the classifier cannot adjudicate — it settles shell command lines through the deny composite, and no hunk touches that composite. The guard: the _VALIDATE_REFUSALS exact-key lookup at the top of conclude_lane's no-report branch, plus validate_refusal_remedy. The call path: fork lane validate step writes rc=corpus-credential|corpus-uncheckable to $GITHUB_OUTPUT and exits 1 → decide step passes it as conclude … --refusal "${VALIDATE_RC:-}" → exact-match lookup selects the named why and remedy= line → publish step carries REMEDY into the check-run summary. Candidate legitimate inputs I could not get a verdict for, in prose: (1) unnamed codes — "", "1", "2", "3", "missing", near-miss spellings like "corpus" and "CORPUS-CREDENTIAL" — must keep reaching the generic and nothing-new rows unchanged; (2) the flipped combination nothing_new=True, marker_present=True, refusal=<named>, which at base read green nothing-new and at head reads unsettled — the diff argues this pair is unreachable (validate refuses only on its rc=0 path, and nothing-new is the exit-3 path, which the base workflow's exit 1 after each rc=corpus-* write supports), and where it would fire it fails closed rather than green; (3) the publish step's generic summary, which no longer points readers at a PR comment that the refusal path never posts. The diff's own unit tests (TestConcludeNamesTheValidateRefusalCause, the emitted-codes drift test) cover these inputs in pytest, but that is not this lane's classifier, so no adjudicated verdict exists for the tightening — which is the gap this verdict names. The narrowing never refuses a user or agent operation; its blast radius is confined to which sentence an already-failed-closed check-run carries.

[SCOPE-REVIEWED] 232a0ba

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 25, 2026
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Board disclosure: the one red is a floored rate-limit casualty, not a verdict

PR Readiness reports 1 blocking readiness item(s). Its own lane tally for head
7b6aa803db08963d3f820581d1da345779be3b8a is:

passed=[CI Fast Gate Build Code Review CodeQL Internal Content Scan
        Opus 5 Review GPT 5.6 Review Design Review UX Review First Principles Review]
pending=[] failed=[Security Scope Review (failure)] skipped=[] awaiting_approval=[]

Security Scope Review is red because of the per-head monotonic floor, not because it
measured anything against this diff. Sequence:

  1. At 02:26-02:28Z the whole board hit the GitHub installation API rate limit. Build / Resolve desktop matrix named it verbatim; the review lanes died in Resolve human override on "Could not read this PR's comments after 3 attempts"; the scope lane
    published failure reading "the scope harness is absent at both refs, so no verdict
    could be folded". Other open PRs were failing the same way in ~80s at the same minute.
  2. The window recovered by 03:10Z (sibling PRs' review lanes completing in 570-810s). Every
    throttled lane was re-run and passed.
  3. The scope lane's re-run at 03:47Z computed a passing verdict and was then held by the
    floor: "held at the harder verdict a prior run of this head published (a prior run of
    this head stands at 'failure', harder than this run's 'success')"
    .

The floor is per-SHA and, on the same-repo lane, cannot discount the earlier row: both rows
are plain github-actions job check-runs with no author-set title, so neither carries the
[scope-floor:unsettled] stamp the fork lane uses for exactly this case. The workflow's own
comment records the asymmetry -- on this lane "a flake there clears only via the override".
Re-running cannot clear it; the floor's message names re-measure-and-push or a human
override for this head.

No lane is a branch-protection gate here (required_status_checks.contexts is []) and
mergeable is true. Awaiting a maintainer decision on which remedy to apply; this branch
is not being pushed in the meantime.

A `validate` refusal dispatches no adjudication leg, so the fold reads
`no-report` and the shared table answers with the generic sentence for that
row. The refusal already writes a code that names its own cause, as a job
output `publish` reads; the table now takes it as an input and selects the
cause for the check-run title plus the action that clears it for the summary.

The conclusion and the `settled` answer are unchanged for every row: a named
refusal selects a more specific `why`, never a softer verdict, and the row
sits ahead of the green nothing-new reading so the two can only fail closed.
Both halves are authored in the table and keyed by exact match, so no value
of the refusal code reaches a published verdict and an unrecognized code
falls through to the generic sentence.

The check-run summary no longer sends a reader to a PR comment for confirmed
rows on a run that folded none. The comment pointer stays in the arm that has
a deterministic body to point at.

Closes #13617
@chenmingwei23
chenmingwei23 force-pushed the fix/security-review-refusal-cause-13617 branch from 7b6aa80 to 232a0ba Compare September 30, 2026 15:53
@chenmingwei23
chenmingwei23 requested a review from a team as a code owner September 30, 2026 15:53
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 30, 2026
@github-actions github-actions Bot added the readiness: passed Eligible automated validation passed for the current revision label Sep 30, 2026

@iamwhatever iamwhatever left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed. No permissions: block touched (publish keeps actions:read, checks:write, contents:read, pull-requests:write), no new secrets.* reference, trigger stays workflow_run so the default-branch definition runs, and scripts/scope_candidates.py runs from the default-branch checkout with persist-credentials false and no ref. VALIDATE_RC is only the base-authored rc=corpus-* echoes matched by exact key against the script's own table, so no caller string reaches GITHUB_OUTPUT. Coupling test pins the workflow's rc codes to the table keys.

@iamwhatever
iamwhatever enabled auto-merge (squash) September 30, 2026 17:37
@iamwhatever
iamwhatever merged commit 2d4278a into main Sep 30, 2026
144 of 145 checks passed
@iamwhatever
iamwhatever deleted the fix/security-review-refusal-cause-13617 branch September 30, 2026 17:40
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Sep 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ci: fork Security Scope Review reports 'no leg reported' without the validate refusal cause

2 participants