Skip to content

ci: move the ten reviewer lanes off the copied perl scrub onto the base-staged scope_redact.py - #22035

Merged
chenmingwei23 merged 1 commit into
mainfrom
ci/10037-lanes-use-scope-redact
Oct 19, 2026
Merged

chenmingwei23 merged 1 commit into
mainfrom
ci/10037-lanes-use-scope-redact

Conversation

@iamwhatever

@iamwhatever iamwhatever commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Every reviewer lane scrubs the text it publishes through the base-owned scripts/scope_redact.py, and no copied perl -i -pe scrub is left under .github/workflows/.

Ten reviewer workflows each carried a copied perl -i -pe credential scrub (#10037). A copy that is wrong is wrong on every surface it guards, and a program inside a run: body cannot be tested. One copy already ate a JSON string's closing quote. #21795 put the two rules only the copies had (GitHub tokens, exact credential values) into the script, so the lanes can now move without losing any redaction.

Why it matters

These lanes publish model-written text to public PR comments, job logs and artifacts, next to live AWS and GitHub credentials. With one tested scrubber, a fix lands once and a test sees it; with sixteen copies, nothing does.

Not a goal

What changed (motivation → approach → change)

Before After
Scrub 16 copied perl -i -pe lines python3 "$REDACTOR" --mode text <file>
Program source inline in the workflow git show "$BASE_SHA:scripts/scope_redact.py", staged once per lane
Scrub fails or script missing perl error, file kept file removed, step fails
Fork GPT and Opus credential values a CRED_VALUE perl loop --redact-env for the three AWS variables

Each lane gains one Stage the base-owned redactor step before its first scrub, the way the Security Scope Review lanes stage theirs. The same-repo lanes check out at depth 2, so each one fetches base.sha first; design-review.yml and ux-review.yml now call need "$BASE_SHA" like the other three.

Every lane gets the script's full rule set. For most that is the same rules plus GitHub tokens. The Codex first pass (codex-review-output.md in codex-review.yml) had only the key-id and assignment rules, so it also gains the ARN and 12-digit account rules. The change under review therefore cannot supply the program that scrubs its own output. Every scrubbed file here is prose, so every call uses --mode text. The [REDACTED-...] marker spellings are unchanged; test/test_scope_redact.py already pins them.

Tests that run a lane's step now hand it the real script: _step_env resolves the staged path to scripts/scope_redact.py, and the lane table marks those lanes needs_redactor instead of needs_perl. A new pin file, test/test_ai_review_workflows_lane_redactor.py, holds the shape: no copy left, staged from the base, every caller after the staging step and reading it, every call refuses, and the two credential lanes pass all three --redact-env names.

For the maintainer to decide

  1. Exit 11 on a token in a corpus field. Since feat(ci): scope_redact.py learns GitHub tokens and --redact-env for the reviewer lanes #21795 the script also matches GitHub tokens. In the Security Scope Review lanes' corpus mode, a token in a field the classifier reads now refuses the run (exit 11) instead of measuring that field. That only redacts more, but it is a behaviour change you may want to rule on.
  2. The perl -0777 -pe error-note scrub. It stays as it is (see Not a goal). Say if you want it moved onto the script in a follow-up issue.

Two low-severity notes on scripts/scope_redact.py itself, which this PR does not change. Local review raised them; say if either should become an issue:

  • --mode text scrubs the whole file in one pass, so the assignment rule's \s* can cross a newline. A line ending in x-amz-security-token: would then rewrite the start of the next line. That fails closed (a marker grep misses), it does not leak.
  • The ARN rule stops at ,, so a role name like role/a,b keeps ,b. The account id is still redacted.

Backwards compatibility

Compatible: the same files are scrubbed at the same steps with the same markers; a scrub that cannot run now fails the step instead of publishing the file.

Removes nothing: the copied perl lines are replaced by a call that applies the same rules and more, and no published surface or output field goes away.

Tests

  • test/test_ai_review_workflows_lane_redactor.py (new): the five pins above across all ten lanes, plus one per same-repo lane that the base commit is fetched before staging (red on design and UX without their need "$BASE_SHA").
  • Existing step-running tests (adjudication, Opus discovery capture, verdict publication, credential values, stamp order, CONCERNS summary) now run against the real script instead of perl.

Determinism: the new file passed 5 repeated runs and one shuffled-order run (shuf of its node ids, -o addopts=""). It reads workflow files only.

Scoped runs: test/test_ai_review_workflows*.py and test/test_scope_redact*.py, 1608 passed. zizmor on the ten workflows reports the same findings as on main (5 high, 53 suppressed).

Manual verification

N/A — unit coverage sufficient: the tests execute the changed run: bodies with the real script, including the refusal branch.

Related Issues

Closes #10037

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@github-actions

github-actions Bot commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision df9c6d484d59e256e643624344e50c235ce63685 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Advisory — does not block merge.

@github-actions

github-actions Bot commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of df9c6d484d59e256e643624344e50c235ce63685 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

Check that git show "$BASE_SHA:scripts/scope_redact.py" resolves in the five fork lanes. No test covers it, and if it fails, those lanes stop publishing.

What this change ships

Inventory (7 items) — 7 justified

Intent: make every reviewer lane redact its published text through one tested, base-owned scrubber instead of 16 copied perl lines. This is a FIX (closes #10037).

  1. The 16 copied perl -i -pe scrubs in ten lanes now call scope_redact.py --mode text — justified
  2. Each lane stages the redactor from the base commit, so the change under review cannot supply its own scrubber — justified
  3. A missing redactor or a failed scrub now deletes the output file and fails the step, where perl's error left the file in place — justified
  4. The Codex first pass gains the ARN and 12-digit account rules, and every lane gains GitHub-token redaction — justified
  5. The fork GPT and Opus lanes swap their perl CRED_VALUE loop for --redact-env on the three AWS variables — justified
  6. design-review.yml and ux-review.yml now fetch base.sha (need "$BASE_SHA") before staging — justified
  7. Step tests run the real script instead of perl, and a new pin file checks the lane shape — justified

[FIRST-PRINCIPLES-REVIEWED] df9c6d4

Description read

[DESCRIPTION-READ] 542432764aad33cb3a927504d2d2bc64acf1be09b19d299d6926fc005d3f975c

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of df9c6d484d59e256e643624344e50c235ce63685 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Check that a failing staged redactor (exit 1) blocks readiness on every lane at once, rather than leaving a lane silently missing.

[DESIGN-REVIEWED] df9c6d4

Description read

[DESCRIPTION-READ] f76298f685d8cb791fe7331eaeef14baab73b7437ca56f87add96fe01e62ea13

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 19, 2026
@github-actions

github-actions Bot commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of df9c6d484d59e256e643624344e50c235ce63685 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] df9c6d4

False positive or not applicable? A repository writer can comment:
/ai-review override gpt df9c6d484d59e256e643624344e50c235ce63685: <one-sentence reason>

@github-actions

github-actions Bot commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed df9c6d484d59e256e643624344e50c235ce63685 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] df9c6d4

Verdict parsed from the review's SHA-scoped output markers for commit df9c6d484d59e256e643624344e50c235ce63685.

False positive or not applicable? A repository writer can comment:
/ai-review override fable df9c6d484d59e256e643624344e50c235ce63685: <one-sentence reason>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 19, 2026
@iamwhatever
iamwhatever force-pushed the ci/10037-lanes-use-scope-redact branch from 6dba247 to 5923a60 Compare October 19, 2026 03:30
@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • span=0cf4b26963b9 fixed in 5923a60

design-review.yml and ux-review.yml now call need "$BASE_SHA" in their shallow-history fetch step, as the other three same-repo lanes already did, so the staging step's git show "$BASE_SHA:..." has the commit.
test_a_shallow_lane_fetches_the_base_commit_before_staging pins it for all five same-repo lanes; it is red on design and UX without the line.

@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • span=b7575d4f36f4 fixed in 5923a60

The body now says it: the Codex first pass (codex-review-output.md) had only the key-id and assignment rules, and gains the ARN and 12-digit account rules. Every other lane gains GitHub tokens only.

@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • span=a6115be27566 fixed in 5923a60

design-review.yml fetches base.sha with need "$BASE_SHA" before the redactor is staged. A failed fetch exits 1; staging then refuses on an empty file and the post step removes the body, so nothing raw is published.

@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • answered: every base these lanes run on carries the script with --redact-env

scripts/scope_redact.py with --redact-env is on main since cc775da (#21795), and this PR's base d32ab41aebbc includes it.
A base without it makes the staging step refuse and the lane fail closed; it never publishes unscrubbed text.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 19, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • span=d737cfb4f099 rebutted: a shared staging helper would be the change under review's own program

The same-repo lanes check out the PR's merge ref, so a .github/scripts/ helper they source is PR-controlled. A helper deciding where the redactor comes from would let the change under review pick its own scrubber, which is what base staging exists to stop.
The ten blocks cannot drift silently: test_the_redactor_is_staged_from_the_base_commit and test_every_scrubbing_step_runs_after_staging_and_reads_the_staged_copy pin every lane. The perl copies had no such test.

@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • answered: the base already carries the script, so first runs stage it

scripts/scope_redact.py with --redact-env has been on main since cc775da. Every PR based on main after that stages it; this PR's own lanes do too (UX Review on this head passed through the staged copy).
A base without it refuses closed with "missing or empty on the base commit"; it never publishes raw text.

@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • answered: the copied part is a fixed fetch, not the rules, and it is pinned

What #10037 removed was sixteen copies of the redaction RULES, which drifted and could not be tested. What remains per lane is a ten-line git show "$BASE_SHA:..." fetch with a refusal; the rules live once, in scripts/scope_redact.py.
It is not moved into a sourced helper on purpose: in the same-repo lanes that helper would come from the PR's own tree, so the change under review could choose its scrubber. Tests pin all ten blocks, so a fix that misses one goes red.

@iamwhatever

Copy link
Copy Markdown
Contributor Author

Rebased: main moved in .github/workflows/ (macos-on-demand.yml) since this head went green (d32ab41 -> 207ed5a); CI re-running.

@iamwhatever
iamwhatever force-pushed the ci/10037-lanes-use-scope-redact branch from 5923a60 to df9c6d4 Compare October 19, 2026 04:22
@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • answered: a failing staged redactor fails each lane's job, it does not go silent

The staging step and every scrub call end in exit 1 on failure, so the job concludes failure and shows red in the rollup like any other failed lane; the scrubbed file is removed first, so nothing is posted.
Each lane stages its own copy from the same base commit, so a broken base script reds all ten lanes on the same run rather than one.

@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • answered: in the fork lanes $BASE_SHA is the checked-out commit itself

All five fork lanes check out ref: ${{ steps.pr.outputs.base_sha }} (step 4), and their staging step reads BASE_SHA: ${{ steps.pr.outputs.base_sha }}. git show of the HEAD commit's own tree cannot miss the object, even at depth 1.
If the base lacks the script, staging refuses with exit 1 and the lane fails closed rather than publishing; test_the_redactor_is_staged_from_the_base_commit pins that refusal in all five.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 19, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor Author

Review-ready at df9c6d4: PR Readiness passed, all 103 checks green, and every Design, First Principles and Opus item has a disposition. Waiting only on a maintainer approval.

Two calls are left for the maintainer, both in the description under For the maintainer to decide: the exit 11 on a GitHub token in a classified corpus field, and whether the separate perl -0777 -pe error-note scrub should move onto the script in a follow-up.

@chenmingwei23
chenmingwei23 merged commit 2b0ef25 into main Oct 19, 2026
103 of 104 checks passed
@chenmingwei23
chenmingwei23 deleted the ci/10037-lanes-use-scope-redact branch October 19, 2026 14:09
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Migrate the ten reviewer lanes off the copied perl scrub to scripts/scope_redact.py

2 participants