Skip to content

feat(ci): scope_redact.py learns GitHub tokens and --redact-env for the reviewer lanes - #21795

Merged
iamwhatever merged 1 commit into
mainfrom
fix/10037-scope-redact-lanes
Oct 19, 2026
Merged

iamwhatever merged 1 commit into
mainfrom
fix/10037-scope-redact-lanes

Conversation

@iamwhatever

@iamwhatever iamwhatever commented Oct 18, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Give scripts/scope_redact.py the two rules the reviewer lanes' copied perl scrub has and it lacks, so those lanes can move onto it in a follow-up PR without losing any redaction.

Ten reviewer workflows scrub their output with a copied perl -i -pe one-liner (#10037). The issue asks them to call the base-owned scripts/scope_redact.py instead. Two copies do things the script cannot do yet:

  • first-principles-review.yml and fork-first-principles-review.yml also redact GitHub token shapes (ghp_..., github_pat_...) to [REDACTED-GH-TOKEN].
  • fork-gpt-review.yml and fork-opus-review.yml also replace the exact inherited AWS credential values with [REDACTED-CREDENTIAL], because a bare secret access key has no shape any rule can see.

Why it matters

The lanes must stage the redactor from the BASE commit, so a lane cannot call a flag the base copy does not have yet. If the migration and these rules landed in one PR, every migrated lane would refuse on that PR. This PR puts the rules on main first; the migration PR then runs on a base that has them.

Not a goal

  • Migrating the ten workflows. That is the follow-up PR, and it is the one that finishes the issue.
  • Any change to the existing rules, markers, exit codes or --drop-changed-fields.

What changed (motivation → approach → change)

Rule Marker Where it applies
GitHub token shapes (gh[pousr]_, github_pat_) [REDACTED-GH-TOKEN] always, both modes
--redact-env NAME (repeatable): the exact value of env var NAME [REDACTED-CREDENTIAL] when asked, both modes

--redact-env takes the variable's name, so the value never appears on a command line. Unset values and values under 16 characters are skipped, like the perl copy's length guard. Exact values are replaced before any shape rule runs. The perl copy did it the other way round, so a 12-digit run inside a session token (between / and +) became [REDACTED-ACCT] first and the rest of the token no longer matched. --redact-env is refused with --drop-changed-fields, because a corpus field is measured verbatim.

Two small fixes ride along, because the new rules make them reachable more often. The assignment rule leaves a whole marker alone, so aws_secret_access_key=<value> ends as =[REDACTED-CREDENTIAL] instead of =[REDACTED]]. A JSON key that carries a credential is still refused, but the error no longer prints the key, since stderr reaches a public job log.

The GitHub token rule also reaches the two Security Scope Review lanes, which already run this script. That only redacts more: a token in prose is redacted, and one in a classified corpus field is refused (exit 11) instead of being measured.

Backwards compatibility

Compatible: every existing rule, marker and exit code is unchanged; the new flag is opt-in, and the new token rule only rewrites text that was being published raw.

Removes nothing: no rule, flag, marker or output field is removed.

Tests

test/test_scope_redact.py:

  • GitHub token shapes redact in text mode, and both markers join the marker-spelling pin the fail-closed greps depend on.
  • --redact-env redacts a bare value in text mode and inside a JSON string, matches before a shape rule can split the value, skips unset and short values, and is refused with --drop-changed-fields.
  • [REDACTED-CREDENTIAL] is pinned as an exact spelling.
  • An assignment keeps an earlier rule's whole marker, and a value that only starts like a marker ([redacted-..., [REDACTEDabc) is still redacted.
  • A refused JSON key is absent from stdout and stderr.

Determinism: all 56 tests in test_scope_redact.py passed 5 repeated runs and one shuffled-order run (shuf of the node ids, -o addopts=""). They use only tmp_path, capsys and monkeypatch.

Scoped runs: test_scope_redact.py, test_scope_redact_classified_pin.py, test_ai_review_workflows_scope_isolation.py, test_ai_review_workflows_verdict_publication.py, test_ai_review_workflows_scope_verdict.py: all pass.

Manual verification

N/A — unit coverage sufficient: the change is a pure text transform and its CLI, both exercised directly.

Related Issues

Part of #10037

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@github-actions

github-actions Bot commented Oct 18, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 612677d55ac36350e2b7a378d14d10efb03a31c5 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Advisory — does not block merge.

@github-actions

github-actions Bot commented Oct 18, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 612677d55ac36350e2b7a378d14d10efb03a31c5 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Nothing to check.

[DESIGN-REVIEWED] 612677d

Description read

[DESCRIPTION-READ] efb3efb8434cea6f9d44eaf56cce87166bdb5c672b0bd6575e3edf3d24663fbb

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 18, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of 612677d55ac36350e2b7a378d14d10efb03a31c5 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

--redact-env has no caller until the follow-up migration PR merges. Check that the PR is actually queued, or the flag stays unused surface.

What this change ships

Inventory (6 items) — 6 justified

Intent: let the ten reviewer lanes drop their copied perl scrub for the base-owned scope_redact.py without losing any redaction (#10037). This is an ADDITION that has to land on the base before the migration can use it.

  1. The redactor now always rewrites GitHub token shapes (gh[pousr]_, github_pat_) to [REDACTED-GH-TOKEN]. — justified
  2. A new repeatable --redact-env NAME flag replaces the exact value of an env var with [REDACTED-CREDENTIAL]. Unset values and values under 16 characters are skipped. — justified
  3. Exact credential values are replaced before any shape rule runs, so an account-id match can no longer split a session token. — justified
  4. --redact-env together with --drop-changed-fields is refused with argparse exit 2. — justified
  5. The assignment rule now leaves an existing [REDACTED-...] marker whole instead of turning it into [REDACTED]]. — justified
  6. A JSON key that carries a credential no longer appears in the stderr refusal message. — justified

[FIRST-PRINCIPLES-REVIEWED] 612677d

Description read

[DESCRIPTION-READ] 2e6de0ccee003a487993541426e7284a0f7c7f1eeff9d50667830208948a2a87

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 18, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 612677d55ac36350e2b7a378d14d10efb03a31c5 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 612677d

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 612677d55ac36350e2b7a378d14d10efb03a31c5: <one-sentence reason>

@github-actions

github-actions Bot commented Oct 18, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 612677d55ac36350e2b7a378d14d10efb03a31c5 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 612677d

Verdict parsed from the review's SHA-scoped output markers for commit 612677d55ac36350e2b7a378d14d10efb03a31c5.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 612677d55ac36350e2b7a378d14d10efb03a31c5: <one-sentence reason>

@github-actions

github-actions Bot commented Oct 18, 2026 •

Copy link
Copy Markdown
Contributor

Security Scope Review -- 🟡 CONCERNS

Which legitimate operations does 612677d55ac36350e2b7a378d14d10efb03a31c5 newly refuse? Updated in place on each push. A script-confirmed regression blocks; the model proposes candidates and never decides a refusal.

Verdict basis: the reviewer's own judgement; no confirmed regression.

Security scope review -- adjudicated candidates

  • macos-latest (platform posix): 8 classified, 0 newly refused, 7 still allowed.
  • ubuntu-latest (platform posix): 8 classified, 0 newly refused, 7 still allowed.
  • windows-latest (platform windows): 6 classified, 0 newly refused, 5 still allowed.

No candidate flipped from allowed to refused.

Scope-Verdict: CONCERNS

TIGHTENING: The outbound scrub scripts/scope_redact.py now redacts more text. It adds GitHub-token shape rules and an exact-value rule driven by environment variables. -- The ONE threat it refuses: a GitHub token, or a bare AWS secret with no prefix, getting into a published review body or rows file. From the hunk: +_GH_TOKEN_RES = ( re.compile(r"\bgh[pousr]_[A-Za-z0-9]{20,}\b"), re.compile(r"\bgithub_pat_[A-Za-z0-9_]{20,}\b"), ) and + for literal in literals: text = text.replace(literal, _CREDENTIAL_MARKER) (fed by --redact-env NAME, values of at least _MIN_LITERAL_LEN = 16 characters). The _SECRET_ASSIGN_RE change adds a lookahead that leaves an existing [REDACTED-...] marker alone. That makes it looser, not stricter.

ADJUDICATED: 11 candidates, 0 confirmed newly-refused
The diff does not touch the shell deny composite (sensitive-bash, exfil, deny-rules). These rows check that the operations around the redactor (running it, testing it, reading GitHub auth state and non-secret AWS variables) still pass that composite. They do not measure the tightening itself.

PLATFORM COVERAGE: I wrote candidates for posix, windows and any.

  • posix: python3 scripts/scope_redact.py ..., pytest test/..., printenv AWS_REGION. These cover the forward-slash path and $VAR spelling the workflows use to run the script.
  • windows: python scripts\scope_redact.py ..., pytest test\..., echo %AWS_REGION%. These cover the backslash path, the python launcher name and %VAR% spelling for a maintainer reproducing the scrub.
  • any: the gh auth status, gh secret list and gh run view rows, whose spelling is the same on every host.

The diff's only new platform assumption is os.environ.get(name). That lookup is case-sensitive on posix and case-insensitive on Windows, so --redact-env aws_secret_access_key finds the value on Windows but silently skips it on posix. This is a redaction-coverage gap, not a refusal, and the shell classifier cannot see it.

UNADJUDICATED: The guard is redact_text / redact_json_text in scripts/scope_redact.py. This is an outbound redactor, not the deny composite, so the classifier produces no verdict for it. An empty adjudicated set says nothing about it.

Call path: fork-security-scope-review.yml (around lines 1515–1531) runs python3 scripts/scope_redact.py --mode json --fail-if-changed "$ROWS". Exit 10 means "changed", and any changed row is withheld rather than published. The body and stderr are scrubbed with --mode text. security-scope-review.yml takes the scrubber from $BASE_SHA, falling back to the checkout or head copy only during bootstrap. Once this change merges, the new rules apply to every later scope review's published rows.

Legitimate inputs I would have submitted for a verdict (prose, since they are rows that would be withheld, not shell refusals):

  • github_pat_ overmatch (the narrowing concern). The class [A-Za-z0-9_]{20,} allows underscores, so ordinary snake_case identifiers starting with github_pat_ match. For example github_pat_expiration_reminder, github_pat_rotation_workflow, or a workflow file github_pat_scopes_check.yml. A golden-path or regression row naming such a job, file or variable would be rewritten to [REDACTED-GH-TOKEN] and withheld. Narrow it: require a real fine-grained PAT shape, such as github_pat_[A-Za-z0-9]{22}_[A-Za-z0-9]{59}, or at least a minimum run of alphanumerics with no underscore after the prefix.
  • gh[pousr]_ prefix. A 20+ character run of letters and digits right after ghs_, ghr_ and the like is unusual in legitimate commands. I see no concrete loss here.
  • --redact-env literal replacement. This is a plain str.replace with no word boundary. If a lane passes a NAME whose value is not secret but appears in ordinary text, every row containing that value is withheld. Examples: a 16+ character GITHUB_REPOSITORY like kirodotdev/KiroCrew, a GITHUB_WORKSPACE path, a role ARN. Inputs: rows containing gh pr view 9332 --repo kirodotdev/KiroCrew, or /home/runner/work/KiroCrew/KiroCrew/... paths, when that variable is passed. The risk depends on which NAMEs callers pass, and no caller in this diff passes any yet. Narrow it: document that the flag is only for credential variables, or refuse a fixed list of known non-secret GITHUB_* / RUNNER_* names.
  • Object-key refusal. This now also fires on any JSON key that matches a GitHub-token shape or a --redact-env value, and a refusal means "unscrubbable". Corpus keys are fixed harness field names, so I expect no legitimate loss.

[SCOPE-REVIEWED] 612677d

@iamwhatever

Copy link
Copy Markdown
Contributor Author

Rebased onto main (9d088c8 -> 943a4e5) to pick up #21792. The red test/test_history_bindings.py::test_the_member_set_is_the_frozen_inventory on the previous head was main's (from #21108), not this diff's; it passes locally on the new head b0743a7. CI re-running.

@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • rebutted: the {20,} GitHub-token bound matches the copies this work replaces

The {16,} spellings live in a different scrubber: the perl -0777 -pe block that cleans API error notes (e.g. claude-review.yml note_read_failure). #10037 does not migrate it.
The copies being replaced are the perl -i -pe lines; the only two with GitHub-token rules, first-principles-review.yml:689 and fork-first-principles-review.yml:711, use {20,}.
Those lanes' withhold greps also use {20,}. So {20,} loses no redaction the migrated lanes have today.

@iamwhatever
iamwhatever force-pushed the fix/10037-scope-redact-lanes branch from b0743a7 to 612677d Compare October 18, 2026 08:24
@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • span=4bd83a17f961 fixed in 612677d

The comment on the --redact-env + --drop-changed-fields refusal was wrong: drop mode never rewrites a field, it only removes one.
It now says the pair is unsupported, not dangerous: drop mode picks fields to remove from the shape rules alone, and no caller needs exact values there.
The migration PR passes --redact-env only with --mode text on the two fork credential lanes, never with --drop-changed-fields.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 18, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • answered: the caller is ready and opens as soon as this merges

The migration is built and tested on branch ci/10037-lanes-use-scope-redact (commit 2308a8908fa1): it removes every perl -i -pe copy and calls --redact-env from fork-gpt-review.yml and fork-opus-review.yml.
It cannot open before this merges: those lanes run the redactor staged from the BASE commit, so on its own PR they would refuse until the flag is on main.
It opens as Closes #10037 right after this lands. If it were dropped, the flag should be removed with it.

@iamwhatever

Copy link
Copy Markdown
Contributor Author

Review-ready at 612677d: PR Readiness passed, every check green (the Windows shard 1 flake #20565 passed on its single rerun), and every Design and First Principles item has a disposition. Waiting only on a maintainer approval. The follow-up PR that moves the ten lanes and closes #10037 opens once this merges.

@chenmingwei23 chenmingwei23 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving per maintainer bulk readiness pass (readiness: passed, no failing lanes).

@iamwhatever
iamwhatever merged commit cc775da into main Oct 19, 2026
153 of 155 checks passed
@iamwhatever
iamwhatever deleted the fix/10037-scope-redact-lanes branch October 19, 2026 02:15
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants