Repository navigation
ci(lint): run hosted black through bounded_black - #13698
Conversation
Every fork PR's Backend Lint job died in the black step with exit 143 after 6-9 minutes: fork PRs fall back to GitHub-hosted ubuntu-latest, where check_black_formatting.py still launched native `python -m black`. The compiled Black wheel retains per-worker memory (the failure the fleet already measured), and the hosted VM has no cgroup cap, so the runner itself was torn down instead of a worker being OOM-killed. Drop the RUNNER_ENVIRONMENT branch: every environment now runs black through scripts/bounded_black.py (one file per spawned worker, at most eight, 2 GiB address space each on Linux). Fork code keeps running on hosted VMs; nothing moves to the CodeBuild fleet. Fixes #13689
|
Intent: Let fork PRs finish the hosted Backend Lint black step by running black through the recycling wrapper on every runner, so the compiled wheel's per-worker retention cannot take down the uncapped |
GPT 5.6 Review — ✅ no blocking findingsGPT 5.6 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
Design Review (Fable 5) — ✅ PASSDesign-level review of Design-Verdict: PASS Verify the first fork-PR hosted lint run finishes green inside the 25-minute budget — this PR's own fleet run never exercises the changed path. [DESIGN-REVIEWED] a4d8d27 |
First Principles Review (Fable 5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Verify the hosted lint job on this PR's own run finished green within the 25-minute budget — the author's 1.6x wall-time numbers came from pure-Python black on arm64, not the compiled wheel. What this change shipsInventory (5 items) — 5 justifiedIntent: stop every fork PR's lint job dying with exit 143 by bounding black's worker memory on hosted runners — a FIX.
[FIRST-PRINCIPLES-REVIEWED] a4d8d27 |
Opus 5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
|
Consolidation: PR #13745 ships the same hosted-black launcher fix; suggest this PR hosts it PR #13745 (fork) fixes the same exit-143 launcher = [str(Path(__file__).with_name("bounded_black.py"))]Both also rewrite the same expectation in Only here: the Only in PR #13745:
Suggestion: land this PR first, as the host. Before merge, change the closing line to From a full open-PR audit against origin/main c66f95b (2026-09-25); findings were checked by independent reviewers reading the code. Reply here if this is wrong. |
Problem / Motivation
Every fork PR's
Backend Lint & Type Check (3.12)job dies in theCheck formatting (black, baselined)step after 6-9 minutes with exit 143 andThe runner has received a shutdown signal. Same-repo PRs andmainare unaffected. #11121 hit it on four consecutive heads; two other fork PRs died the same way in the same hour.Why it matters
No fork PR can reach a green
PR Readiness, so every external contribution needs a maintainer override on Lint.What changed (motivation → approach → change)
scripts/bounded_black.pyscripts/bounded_black.pyRUNNER_ENVIRONMENT=github-hosted, every fork PR)python -m black, default poolscripts/bounded_black.pyThe lint job runs on
${{ needs.changes.outputs.linux_runner_large || 'ubuntu-latest' }}. Same-repo PRs resolve to the CodeBuild fleet; fork PRs fall back to hostedubuntu-latest.scripts/check_black_formatting.pychose its launcher byRUNNER_ENVIRONMENT: the fleet and local runs went throughbounded_black.py, hosted kept the native command.Hosted kept native black because the fleet migration (#11880) only had a measured failure on the fleet, whose cgroup OOM-killed a two-worker native pool at 15 GB, and left hosted behaviour untouched as the conservative choice. That reasoning no longer holds. The failed hosted job's own diagnostics show
memory.max: max(no cgroup cap) on a 16 GB host, so the same per-worker retention exhausts the VM instead of one worker: the runner service is torn down and the step ends with SIGTERM (143). The retention is a property of the compiled Black wheel --black-26.3.1-cp312-cp312-manylinux_*_x86_64.whlships 30 mypyc.somodules and is exactly whatuv pip installpicks onubuntu-latest-- which is why the issue's local repro on a pure-Python black stayed under 1 GB and passed.The fix drops the
RUNNER_ENVIRONMENTbranch inscripts/check_black_formatting.py: every environment launches black throughscripts/bounded_black.py(one file per spawned worker, at most eight workers, 2 GiB address space per process on Linux). Native discovery, configuration, cache and verdict semantics are unchanged; incomplete reports still return 123. The.github/workflows/ci.ymlstep comment anddocs/ci/ci-and-reviews.mddescribe the single path, andtest/test_black_fleet_budget.pypins thatgithub-hostedselects the wrapper too. Fork code stays on GitHub-hosted VMs; nothing moves to the CodeBuild fleet.Cost: recycling is about 1.6x native wall time on four cores (numbers below), so the hosted black step should land around 12 minutes, inside the job's 25-minute budget. If a hosted run shows otherwise,
timeout-minutesis the knob, not the launcher.Backwards compatibility
Compatible: the gate's inputs, scope, baseline and exit codes are unchanged; only the process that runs black on hosted runners differs. Local runs already used the wrapper.
Tests
test/test_black_fleet_budget.py::test_gate_preserves_scope_workers_and_baseline_on_black_result(15 cases): everyRUNNER_ENVIRONMENTvalue,github-hostedincluded, launchesscripts/bounded_black.pywith the same arguments, inheritsBLACK_NUM_WORKERS, and preserves scope, baseline and black's exit code.test/test_black_fleet_budget.py::test_two_worker_real_gate_preserves_ratchet[github-hosted]now exercises the real wrapper with black's default pool (noBLACK_NUM_WORKERS) and still keeps offender / graduate / parse-error verdicts.test/test_black_fleet_budget.py,test/test_black_gate_contract.py,test/test_prepare_pr_profiles.py-- 140 passed.Manual verification
Ran the workflow step as written,
RUNNER_ENVIRONMENT=github-hosted python3 scripts/ci_black_diagnostics.py scripts/check_black_formatting.py, coldBLACK_CACHE_DIR, pinned to 4 cores (taskset -c 0-3) to matchubuntu-latest: gate exit 0, wall 472 s, peak child RSS 349 MiB,oom_kill 0. Launcher comparison on the same tree (4690 files, 4 workers, cold cache): native 284 s; bounded 451 s unpinned, 457 s pinned to 4 cores; bounded with 8 workers on 4 cores 478 s, so the default pool stays. These numbers come from a pure-Python black on arm64 -- they measure the recycling overhead, not the compiled-wheel retention, which only the hosted CI run on this PR can show.Review: model-pinned local reviewers were not dispatched -- this session's governance forbids creating sub-agents -- so the change was self-reviewed against the GPT and Opus charters (AUTOSDE blocking rules checked: none match; the test change tightens, it does not loosen).
Related Issues
Motivating case: #11121.
Fixes #13689
Pattern harvest
Rule candidate: review-prompt
Pattern: an environment-conditional launcher keeps the unbounded path on the environment nobody measured; a bound proven on one runner class should be the default everywhere unless the other class has its own guard.
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)