Skip to content

ci(ratchet-audit): carry Black's cache between main pushes - #30280

Merged
iamwhatever merged 1 commit into
mainfrom
ci/ratchet-audit-black-cache
Nov 5, 2026
Merged

iamwhatever merged 1 commit into
mainfrom
ci/ratchet-audit-black-cache

Conversation

@tim-finnigan

@tim-finnigan tim-finnigan commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Main gets its ratchet verdict sooner after each merge, because the whole-tree black check in Main Ratchet Audit stops re-checking files that have not changed since the last push.

The Check formatting (black, baselined) step takes a median 17.5 minutes (13.2 to 21.9) of a 23.2-minute job, measured over 16 push runs sampled evenly across 2026-11-04 09:37Z to 2026-11-05 06:28Z (#30259). Install takes 0.1 minutes; no other step takes more than about a minute.

Why it matters

Main Ratchet Audit renders main's ratchet verdict for every push, and the merge queue's required check waits on it. Every minute in this step is a minute before drift on main shows up, or before a merge group can land.

Not a goal

What changed (motivation → approach → change)

Black is not the slow part. bounded_black.py starts a fresh interpreter and imports Black once per file, about 10,940 times a run, on a Black cache that is empty every time. On the same tree locally, the wrapper took 671 s with an empty cache, native black -W 8 took 142 s, and the wrapper with Black's cache warm took 67 s. A warm run only re-checks files whose content changed, plus the 724 baselined files, which Black never caches because it only caches files it found formatted.

Black's cache cannot be saved as-is. It keys each entry on the file's absolute path, and the CodeBuild runner checks out under a new /codebuild/output/src<random>/ root on every build, so a restored cache would match nothing.

So scripts/black_cache_portable.py carries it in a portable form, and Black reads and writes its own file throughout:

  • export reads the cache with Black's black.cache.Cache.read and writes JSON: the cache file's name, and each formatted file's repository-relative path with the SHA-256 Black recorded for it.
  • restore hashes each listed file in the current checkout with Cache.hash_digest, and passes only the files whose bytes still match to Cache.write, which records each file's stat and hash itself.

The script contains no pickle and parses nothing but JSON. The black.Mode it hands Cache.read is the gate's own: the TARGET_VERSION that check_black_formatting.py passes on the command line, plus [tool.black] line-length read with Black's parse_pyproject_toml. Any other [tool.black] key that could change the mode turns the cache off, because this script would not reproduce it.

The workflow restores the JSON (at the relative path .cache/black-gate-cache.json, for the same changing-root reason) with actions/cache/restore before the black step and saves it after, from main pushes only (the same github.event_name == 'push' && github.ref == 'refs/heads/main' condition as ci.yml's mypy cache). Any cache problem is a warning and a cold run, never a failed step. That includes a Black release whose black.cache API is missing, renamed or takes different arguments.

The verdict is the cold run's because the rules that decide reuse are Black's own: a file is handed back only when its current SHA-256 matches the content Black found formatted, the cache file name carries Black's mode key (target versions, line length, preview features) and a saved cache for another mode is refused, and a cache saved by another Black version is refused. The actions/cache key also includes hashFiles('pyproject.toml'), where the Black pin and [tool.black] live, so a pin or config change on main restores nothing. That cost one cold run in the last week (pyproject.toml changed once in 2,551 main commits).

The 40-minute job timeout stays, since a cold cache still pays the full step. Its comment now says so.

Backwards compatibility

Compatible: the gate's command, scope and verdict are unchanged, and a missing or rejected cache only makes the run as slow as it is today.

Removes nothing: the diff adds a cache around an existing step and changes no gate, file or key.

Tests

test/test_black_cache_portable.py, which seeds and reads Black's cache through black.cache.Cache as the script does:

  • the mode is the gate's (py310, line length 100);
  • a cache exported from one checkout restores under a different root, and a file whose bytes changed is not handed back;
  • entries outside the checkout are not exported;
  • a cache saved by another Black version, or for another mode (a line-length change), restores nothing;
  • an unmapped [tool.black] setting such as preview turns the cache off;
  • absolute, .., backslash and empty paths are dropped;
  • a missing, unparsable or unknown-format saved cache restores nothing;
  • a black.cache with no read, a read taking different arguments, a renamed write, a non-dict file_data, or no black.cache module at all: restore and export both exit 0 with an empty cache, and the gate step is untouched;
  • the script contains no pickle;
  • the workflow saves only on main pushes, the cached path is the same literal string on every runner, and the black gate step reads the same BLACK_CACHE_DIR the restore and export steps use, in that order.

On the final head, the new file with test/test_main_ratchet_audit.py and test/test_black_gate_contract.py: 81 passed.

Proof: not provable -- other: CI workflow speed change; the new script has no previous version to revert to

Manual verification

CI, on the CodeBuild runner, via workflow_dispatch on a scratch branch (since deleted). The branch carried head 6d24bad plus a one-line change letting the dispatch save the cache. The script and workflow on it are the same as this PR's; only the test file changed after.

Run Black cache Black step Whole job Gate
42018119772 cold, then 10,246 entries exported and saved 15.7 min 20.6 min passed, 724 baselined
42019089349 warm, 10,246 entries restored 2.6 min 8.2 min passed, 724 baselined
42022849724 warm, plus a scratch commit adding a misformatted file in src/kiro_crew/recovery/ (last changed 2026-09-25) and misformatting the cached, clean src/kiro_crew/providers/mirrors/codex.py (changed today) 2.4 min 7.8 min failed, naming exactly those two files

The third run restored 10,245 entries, not 10,246: codex.py's bytes no longer hashed to the recorded value, so it was not handed back to Black and Black re-checked it.

An earlier version saved the JSON under ${{ runner.temp }}, and the next run missed the exact key it had just saved, because actions/cache folds the path string into the entry's version and runner.temp expands to the build's own root. The relative, gitignored .cache/black-gate-cache.json fixes that, and a test pins it.

Local, black 26.10.1: a real cache for this tree exported 10,219 entries through Cache.read, and restore handed 10,037 back to Black; the other 182 files had changed since that cache was written. The two weakened-guard checks were run by hand: dropping the path check, or the file_data type check, makes the matching tests fail (4 failures).

Related Issues

Fixes #30259

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributions are accepted under the project's Apache-2.0 license; see the Licensing section of CONTRIBUTING.md.

@tim-finnigan
tim-finnigan requested review from a team and KiroCrewCaptain as code owners November 5, 2026 11:06
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 5, 2026
@github-actions

github-actions Bot commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 4878e5f3286cb0505d450f9f2a39d65198a7a530 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of 4878e5f3286cb0505d450f9f2a39d65198a7a530 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Nothing to check.

What this change ships

Inventory (6 items) — 6 justified

Intent: get main's ratchet verdict sooner by not re-running Black on files that haven't changed since the last main push. This is a FIX for a slow step, measured in #30259.

  1. On main pushes, the black step reuses the previous push's "already formatted" results, so only changed or baselined files are checked again — justified
  2. Main pushes now save Black's cache as portable JSON with repository-relative paths and SHA-256 hashes — justified
  3. New script scripts/black_cache_portable.py with export and restore commands, reading and writing through Black's own black.cache.Cache — justified
  4. Any cache problem (a different Black version, a different mode, an unmapped [tool.black] key, an API change) prints a warning and the run starts cold — justified
  5. The comment on the 40-minute job timeout now explains that a cold-cache run still needs the full time — justified
  6. New test file test/test_black_cache_portable.py pins portability, the refusal rules and the save-on-main-only workflow wiring — justified

[FIRST-PRINCIPLES-REVIEWED] 4878e5f

Description read

[DESCRIPTION-READ] 1692133b6d34d97a7524067301127ebd7b96c08c110ff543e8092203191b307d

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 4878e5f3286cb0505d450f9f2a39d65198a7a530 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

No design-level concerns. Reuse is decided entirely by Black's own content hash, mode key and version check. A missing or rejected cache just means a cold run with the same verdict, and the scratch-branch runs show a warm run that still catches the two misformatted files.

Notes

  • Every main push saves a new cache entry keyed by its SHA. The JSON is small and actions/cache evicts old entries, so this needs no pruning step in this PR.

[DESIGN-REVIEWED] 4878e5f

Description read

[DESCRIPTION-READ] b57ba54e9331f67eab3bdc549cb9913b1fb0f056597e66d3cae45cd4c90025d9

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 4878e5f3286cb0505d450f9f2a39d65198a7a530 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 4878e5f

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 4878e5f3286cb0505d450f9f2a39d65198a7a530 false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 4878e5f3286cb0505d450f9f2a39d65198a7a530 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 4878e5f

Verdict parsed from the review's SHA-scoped output markers for commit 4878e5f3286cb0505d450f9f2a39d65198a7a530.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 4878e5f3286cb0505d450f9f2a39d65198a7a530 false-positive: <reason, 20 to 500 characters>

@tim-finnigan
tim-finnigan force-pushed the ci/ratchet-audit-black-cache branch 3 times, most recently from a558349 to 37d5415 Compare November 5, 2026 11:27
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 5, 2026
The whole-tree black step in main-ratchet-audit took a median 17.5 of
23.2 minutes over 16 sampled runs. Almost all of it is bounded_black's
one-process-per-file start-up on an empty Black cache. Black's own cache
keys entries on absolute paths and the CodeBuild checkout root changes
every build, so the cache is exported as repository-relative JSON on
main pushes and re-rooted before the gate runs. Black's verdict is
unchanged: entries are reused only on a matching content hash, and the
mode key and Black version are part of the cache's identity.

Refs #30259
@tim-finnigan
tim-finnigan force-pushed the ci/ratchet-audit-black-cache branch from 37d5415 to 4878e5f Compare November 5, 2026 12:12
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Nov 5, 2026

@iamwhatever iamwhatever left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve at 4878e5f3286cb0505d450f9f2a39d65198a7a530.

The gate's verdict still comes from Black. restore hands Black only files whose current bytes hash to the recorded SHA-256 (black_cache_portable.py L209). It refuses a cache from another Black version or mode (L198-L201), and every failure is a warning followed by a cold run. Only a main push exports and saves the cache (workflow L196-L223), so a merge group can restore main's entry but cannot write one. Workflow permissions stay contents: read, no secrets are added, and actions/cache is pinned to the same SHA that ci.yml uses. The 40-minute cap is kept for cold runs. Checks are green.

AI findings judged non-blocking

  • Design Review (Notes): each main push saves a new cache entry keyed by its SHA. The entries are small JSON files that actions/cache evicts, and the restore-keys prefix picks the newest, so correctness is unaffected.

— crew-pr-reviewer (r146)

@iamwhatever
iamwhatever merged commit b0d6381 into main Nov 5, 2026
97 checks passed
@iamwhatever
iamwhatever deleted the ci/ratchet-audit-black-cache branch November 5, 2026 17:31
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Nov 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ci: main-ratchet-audit's whole-tree black step takes a median 17.5 of 23 minutes, almost all of it per-file process start-up on a cold cache

2 participants