Skip to content

fix(ci): fetch only the two commits internal content scan diffs - #29293

Closed
buluoray wants to merge 1 commit into
mainfrom
crew/slack/ci-checkout-shallow
Closed

buluoray wants to merge 1 commit into
mainfrom
crew/slack/ci-checkout-shallow

Conversation

@buluoray

@buluoray buluoray commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Internal Content Scan stops being cancelled at its 10-minute limit while actions/checkout downloads the whole repository.

On 2026-11-03, since 15:00Z, 17 Internal Content Scan Gate runs were cancelled and none succeeded, across many PRs (for example run 41690485822 on #29261 and run 41684185557 on #29192). Each was cancelled inside actions/checkout at the job's timeout-minutes: 10. That checkout was fetch-depth: 0: every branch and every blob ever committed, about 3 GB (#27513 measured 2.87 GiB).

Why it matters

Internal Content Scan is a required, blocking lane. A cancelled run publishes no verdict, so PR Readiness stays red on PRs whose code is fine, and each hit costs a rerun or a fresh push that restarts every check.

Not a goal

What changed (motivation → approach → change)

Symptom: the scan job is cancelled in checkout. Cause: the job diffs git diff --unified=0 "$BASE" "$HEAD", two-dot, so it reads exactly two trees, yet its checkout downloads the full history with all blobs.

Before After
Checkout fetch-depth: 0 fetch-depth: 2 (the checked-out commit and its parent)
Missing end of the range already in the full clone fetched by commit id, --depth=1, all files, 3 attempts
End that cannot be fetched git diff fails step fails with could not fetch <sha>; failing closed.
Diff git diff --unified=0 BASE HEAD unchanged

Per event shape (the callee serves all three that internal-content-scan-gate.yml delivers):

  • push: HEAD is the checkout; before is fetched by id. A branch's first push falls back to HEAD^, which depth 2 already has.
  • pull_request: HEAD is the merge commit (the checkout); base.sha is fetched by id.
  • merge_group: HEAD is the group head (the checkout); base_sha is fetched by id.

Why the gate stays as strong:

  • The diff command and its range are byte-for-byte unchanged, and two-dot git diff reads only the two commits' trees, so it sees the same lines a full clone gives.
  • No partial-clone filter, so no object is fetched on demand during the diff. This is the hazard fix(ci): fetch only what Code Review's jobs read, not every branch #27513 named for filter: blob:none on diff-reading gates; it does not apply here.
  • Every way an end can be missing fails the step before the diff runs.

The fetch sends the job's own github.token (contents: read) as a masked http.extraheader, the same form fork-internal-content-scan.yml uses, so a private calling repository can still fetch its own commits. Checkout keeps persist-credentials: false.

release.yml is left out on purpose. Its Stable publication gate reads refs/tags/v$VERSION, the tagged tree's version files and CHANGELOG.md, and its promotion check depends on how that tag resolves. Proving that a shallow checkout resolves the tag the same way under every release trigger is a separate, security-sensitive change, and its run 41672019878 passed on attempt 2.

Backwards compatibility

Compatible: the scan reads the same two trees and produces the same diff for every event shape; only how the objects arrive changes.
Removes nothing: no step, output, check or ruleset input is removed; test_workflow_bounded_checkout.py's allowlist loses the entry this job no longer needs.

Tests

  • test/test_internal_content_scan_checkout.py (new): builds a local origin, checks out at depth 2 as actions/checkout does, runs the step's real run: body, and compares change.diff with a full clone's for push, a branch's first push, pull_request and merge_group. An unfetchable end fails the step and writes no diff. The checkout carries no filter.
  • test/test_workflow_bounded_checkout.py: internal-content-scan.yml/scan is no longer allowed to keep fetch-depth: 0 under a tight timeout.
  • Mutation: removing the fetch loop fails the push, pull_request, merge_group and unfetchable cases.
  • Locally: test_workflow_*.py, test_release_*.py, the fleet-routing and watchdog workflow tests: 1012 passed. The 7 errors are test_workflow_injection_identity.py asking for a missing event_loop fixture in this environment, unrelated to workflow YAML.

Proof: PROVEN test/test_internal_content_scan_checkout.py::test_the_checkout_is_two_commits_and_the_diff_step_comes_right_after, test/test_internal_content_scan_checkout.py::test_the_diff_matches_a_full_clone[push], test/test_internal_content_scan_checkout.py::test_the_diff_matches_a_full_clone[pull_request], test/test_internal_content_scan_checkout.py::test_the_diff_matches_a_full_clone[merge_group], test/test_internal_content_scan_checkout.py::test_an_end_that_cannot_be_fetched_fails_the_step (head 8538c9a)

Manual verification

This PR's own Internal Content Scan Gate run exercises the pull_request shape on the real runner. actionlint is not installed here, so it was not run locally.

Related Issues

Related #29264, #28443

Sibling sweep

Swept: git grep -n "fetch-depth: 0" -- .github/workflows
Sibling: .github/workflows/internal-content-scan.yml -- changed here
Sibling: .github/workflows/security-scope-review.yml -- tracked in #29264
Sibling: .github/workflows/fork-security-scope-review.yml -- tracked in #29264
Sibling: .github/workflows/main-ratchet-audit.yml -- tracked in #28443
Sibling: .github/workflows/release.yml -- not affected -- stable gate resolves a tag; left out, see What changed
Sibling: .github/workflows/fork-internal-content-scan.yml -- not affected -- three-dot diff needs the merge-base history
Sibling: .github/workflows/add-contributor.yml -- not affected -- reads git log across merged history

Pattern harvest

Rule candidate: lint
Pattern: a job checks out fetch-depth: 0 when its steps read only two named commits' trees

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributions are accepted under the project's Apache-2.0 license.

The scan diffs BASE..HEAD two-dot, so it reads two trees. Its checkout
fetched full history with every blob, about 3 GB, and on 11-03 every
run since 15:00Z was cancelled in actions/checkout at the job's
10-minute limit. Check out at depth 2 and fetch a missing end by id,
with all of its files, failing closed when it cannot be fetched.
@buluoray
buluoray requested review from a team and KiroCrewCaptain as code owners November 3, 2026 19:42
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 3, 2026
@github-actions

github-actions Bot commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 8538c9ae3982dc6f711115c3b9df3c39dbe96d40 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks. The two-dot diff only reads two trees, both ends are fetched in full with no partial-clone filter, and the step fails closed if either end cannot be fetched.

Notes

  • The removed allowlist comment ("migrating requires event-shaped fetch logic") describes work still to do, not a choice to keep full history, so retiring that entry is consistent with what it pinned.

[DESIGN-REVIEWED] 8538c9a

Description read

[DESCRIPTION-READ] 72c69c03703a2e6da14596a5a22586b2e6a9fbe896050978eebf3077e82802ae

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of 8538c9ae3982dc6f711115c3b9df3c39dbe96d40 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Nothing blocks. Every item comes from the measured checkout cancellations. The deleted allowlist pin is replaced by the event-shaped fetch it said was missing.

Notes

  • Item 4 — inherited: the PR cites no failed one-commit fetch to justify "3 attempts" with sleep $((attempt * 5)). The fail-closed check after the fetch already reports an end that is missing. Subtraction: delete the for attempt in 1 2 3 loop and keep one git fetch.

What this change ships

Inventory (5 items) — 4 justified

Intent: let Internal Content Scan finish within its 10-minute limit instead of being cancelled during a full-history checkout. FIX.

  1. The scan job checks out two commits (fetch-depth: 2) instead of the whole history — justified
  2. The diff step fetches a missing end of the range by commit id, with all its files — justified
  3. A missing or unfetchable end now fails the step with a "failing closed" error — justified
  4. Each fetch is tried up to 3 times with growing sleeps — inherited
  5. internal-content-scan.yml/scan is removed from the bounded-checkout allowlist and from the still-expected set — justified

[FIRST-PRINCIPLES-REVIEWED] 8538c9a

Description read

[DESCRIPTION-READ] 3bcf693a83a629681fa08abe5363428bd48743ed176711a7db4fd2d1d5c60050

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 8538c9ae3982dc6f711115c3b9df3c39dbe96d40 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 8538c9a

Verdict parsed from the review's SHA-scoped output markers for commit 8538c9ae3982dc6f711115c3b9df3c39dbe96d40.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 8538c9ae3982dc6f711115c3b9df3c39dbe96d40 false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 8538c9ae3982dc6f711115c3b9df3c39dbe96d40 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 8538c9a

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 8538c9ae3982dc6f711115c3b9df3c39dbe96d40 false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 8538c9ae3982dc6f711115c3b9df3c39dbe96d40 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@buluoray

buluoray commented Nov 3, 2026

Copy link
Copy Markdown
Contributor Author

Live evidence on head 8538c9a: this PR's own Internal Content Scan Gate run 41705705357 passed. Checkout (depth 2) took 117 s, the diff step under 1 s (range 4edd445...0d75b16, 261 diff lines; the base was the merge commit's first parent, so no extra fetch). In the same window, the ten full-clone runs started 19:34-19:41Z on other branches and main were all cancelled in checkout at 10 minutes. The 117 s for two commits suggests GitHub's fetch is slow today in general; the shallow fetch still fits the budget the full one cannot.

@github-actions github-actions Bot added the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Nov 3, 2026
@buluoray

buluoray commented Nov 3, 2026

Copy link
Copy Markdown
Contributor Author

Closing: #29301 merged at 20:53Z (4e44f73) and now owns this checkout (fetch-depth: 0 + filter: blob:none, 20-minute budget). There was no reply on the overlap question in this comment, so we are not reopening the file. For the record, the depth-2 approach here passed on its own head: checkout took 117 s (run 41705705357), while the ten full-clone scan runs started 19:34-19:41Z were cancelled in checkout at 10 minutes (evidence). If the blobless full clone still runs long, this branch is a ready alternative.

@buluoray buluoray closed this Nov 3, 2026
@github-actions github-actions Bot removed the readiness: checking Automated validation is still running label Nov 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

merge conflict Branch has merge conflicts with its base — author must resolve before merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant