Skip to content

ci(lint): run hosted black through bounded_black - #13698

Merged
iamwhatever merged 1 commit into
mainfrom
fix/fork-lint-bounded-black
Sep 25, 2026
Merged

iamwhatever merged 1 commit into
mainfrom
fix/fork-lint-bounded-black

Conversation

@CrysisDeu

Copy link
Copy Markdown
Contributor

Problem / Motivation

Every fork PR's Backend Lint & Type Check (3.12) job dies in the Check formatting (black, baselined) step after 6-9 minutes with exit 143 and The runner has received a shutdown signal. Same-repo PRs and main are unaffected. #11121 hit it on four consecutive heads; two other fork PRs died the same way in the same hour.

Why it matters

No fork PR can reach a green PR Readiness, so every external contribution needs a maintainer override on Lint.

What changed (motivation → approach → change)

before after
Fleet / local black scripts/bounded_black.py scripts/bounded_black.py
GitHub-hosted black (RUNNER_ENVIRONMENT=github-hosted, every fork PR) native python -m black, default pool scripts/bounded_black.py

The lint job runs on ${{ needs.changes.outputs.linux_runner_large || 'ubuntu-latest' }}. Same-repo PRs resolve to the CodeBuild fleet; fork PRs fall back to hosted ubuntu-latest. scripts/check_black_formatting.py chose its launcher by RUNNER_ENVIRONMENT: the fleet and local runs went through bounded_black.py, hosted kept the native command.

Hosted kept native black because the fleet migration (#11880) only had a measured failure on the fleet, whose cgroup OOM-killed a two-worker native pool at 15 GB, and left hosted behaviour untouched as the conservative choice. That reasoning no longer holds. The failed hosted job's own diagnostics show memory.max: max (no cgroup cap) on a 16 GB host, so the same per-worker retention exhausts the VM instead of one worker: the runner service is torn down and the step ends with SIGTERM (143). The retention is a property of the compiled Black wheel -- black-26.3.1-cp312-cp312-manylinux_*_x86_64.whl ships 30 mypyc .so modules and is exactly what uv pip install picks on ubuntu-latest -- which is why the issue's local repro on a pure-Python black stayed under 1 GB and passed.

The fix drops the RUNNER_ENVIRONMENT branch in scripts/check_black_formatting.py: every environment launches black through scripts/bounded_black.py (one file per spawned worker, at most eight workers, 2 GiB address space per process on Linux). Native discovery, configuration, cache and verdict semantics are unchanged; incomplete reports still return 123. The .github/workflows/ci.yml step comment and docs/ci/ci-and-reviews.md describe the single path, and test/test_black_fleet_budget.py pins that github-hosted selects the wrapper too. Fork code stays on GitHub-hosted VMs; nothing moves to the CodeBuild fleet.

Cost: recycling is about 1.6x native wall time on four cores (numbers below), so the hosted black step should land around 12 minutes, inside the job's 25-minute budget. If a hosted run shows otherwise, timeout-minutes is the knob, not the launcher.

Backwards compatibility

Compatible: the gate's inputs, scope, baseline and exit codes are unchanged; only the process that runs black on hosted runners differs. Local runs already used the wrapper.

Tests

  • test/test_black_fleet_budget.py::test_gate_preserves_scope_workers_and_baseline_on_black_result (15 cases): every RUNNER_ENVIRONMENT value, github-hosted included, launches scripts/bounded_black.py with the same arguments, inherits BLACK_NUM_WORKERS, and preserves scope, baseline and black's exit code.
  • test/test_black_fleet_budget.py::test_two_worker_real_gate_preserves_ratchet[github-hosted] now exercises the real wrapper with black's default pool (no BLACK_NUM_WORKERS) and still keeps offender / graduate / parse-error verdicts.
  • Ran locally: test/test_black_fleet_budget.py, test/test_black_gate_contract.py, test/test_prepare_pr_profiles.py -- 140 passed.

Manual verification

Ran the workflow step as written, RUNNER_ENVIRONMENT=github-hosted python3 scripts/ci_black_diagnostics.py scripts/check_black_formatting.py, cold BLACK_CACHE_DIR, pinned to 4 cores (taskset -c 0-3) to match ubuntu-latest: gate exit 0, wall 472 s, peak child RSS 349 MiB, oom_kill 0. Launcher comparison on the same tree (4690 files, 4 workers, cold cache): native 284 s; bounded 451 s unpinned, 457 s pinned to 4 cores; bounded with 8 workers on 4 cores 478 s, so the default pool stays. These numbers come from a pure-Python black on arm64 -- they measure the recycling overhead, not the compiled-wheel retention, which only the hosted CI run on this PR can show.

Review: model-pinned local reviewers were not dispatched -- this session's governance forbids creating sub-agents -- so the change was self-reviewed against the GPT and Opus charters (AUTOSDE blocking rules checked: none match; the test change tightens, it does not loosen).

Related Issues

Motivating case: #11121.

Fixes #13689

Pattern harvest

Rule candidate: review-prompt
Pattern: an environment-conditional launcher keeps the unbounded path on the environment nobody measured; a bound proven on one runner class should be the default everywhere unless the other class has its own guard.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Every fork PR's Backend Lint job died in the black step with exit 143
after 6-9 minutes: fork PRs fall back to GitHub-hosted ubuntu-latest,
where check_black_formatting.py still launched native `python -m black`.
The compiled Black wheel retains per-worker memory (the failure the
fleet already measured), and the hosted VM has no cgroup cap, so the
runner itself was torn down instead of a worker being OOM-killed.

Drop the RUNNER_ENVIRONMENT branch: every environment now runs black
through scripts/bounded_black.py (one file per spawned worker, at most
eight, 2 GiB address space each on Linux). Fork code keeps running on
hosted VMs; nothing moves to the CodeBuild fleet.

Fixes #13689
@CrysisDeu
CrysisDeu requested review from a team and bolichen97 September 25, 2026 05:48
@CrysisDeu

Copy link
Copy Markdown
Contributor Author

Intent: Let fork PRs finish the hosted Backend Lint black step by running black through the recycling wrapper on every runner, so the compiled wheel's per-worker retention cannot take down the uncapped ubuntu-latest VM.
Not a goal: Moving fork PRs onto the CodeBuild fleet, changing the black baseline/ratchet semantics, or tuning the lint job timeout.

@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Sep 25, 2026
@github-actions

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ✅ no blocking findings

GPT 5.6 completed its review of a4d8d2787c1495d97c4bc0943a4e06552c9b482a and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] a4d8d27

False positive or not applicable? A repository writer can comment:
/ai-review override gpt a4d8d2787c1495d97c4bc0943a4e06552c9b482a: <one-sentence reason>

@github-actions

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — ✅ PASS

Design-level review of a4d8d2787c1495d97c4bc0943a4e06552c9b482a — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Verify the first fork-PR hosted lint run finishes green inside the 25-minute budget — this PR's own fleet run never exercises the changed path.

[DESIGN-REVIEWED] a4d8d27

@github-actions

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5) — ✅ PASS

Premise-level review of a4d8d2787c1495d97c4bc0943a4e06552c9b482a — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

Verify the hosted lint job on this PR's own run finished green within the 25-minute budget — the author's 1.6x wall-time numbers came from pure-Python black on arm64, not the compiled wheel.

What this change ships

Inventory (5 items) — 5 justified

Intent: stop every fork PR's lint job dying with exit 143 by bounding black's worker memory on hosted runners — a FIX.

  1. Fork-PR lint no longer tears down the runner mid-step; hosted black runs recycled, memory-capped workers — justified
  2. The environment-conditional launcher branch is deleted; one launch path everywhere — justified
  3. Hosted black step gets ~1.6x slower (measured, stays inside the job budget) — justified
  4. CI step comment and docs/ci/ci-and-reviews.md now describe the single path — justified
  5. Test pin rewritten: github-hosted now asserts the wrapper, reversing the ci: migrate eligible linux and windows jobs to codebuild #11880 conservative choice with the measured hosted failure as evidence — justified

[FIRST-PRINCIPLES-REVIEWED] a4d8d27

@github-actions

Copy link
Copy Markdown
Contributor

Opus 5 Review — ✅ no blocking findings

Reviewed a4d8d2787c1495d97c4bc0943a4e06552c9b482a — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] a4d8d27

Verdict parsed from the review's SHA-scoped output markers for commit a4d8d2787c1495d97c4bc0943a4e06552c9b482a.

False positive or not applicable? A repository writer can comment:
/ai-review override fable a4d8d2787c1495d97c4bc0943a4e06552c9b482a: <one-sentence reason>

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Sep 25, 2026
@bolichen97

Copy link
Copy Markdown
Contributor

Consolidation: PR #13745 ships the same hosted-black launcher fix; suggest this PR hosts it

PR #13745 (fork) fixes the same exit-143 Check formatting (black, baselined) failure with the same core edit. Both delete the RUNNER_ENVIRONMENT == "github-hosted" -> ["-m", "black"] branch in scripts/check_black_formatting.py::_unformatted and put the identical line in its place (scripts/check_black_formatting.py:100 here):

launcher = [str(Path(__file__).with_name("bounded_black.py"))]

Both also rewrite the same expectation in test_gate_preserves_scope_workers_and_baseline_on_black_result (test/test_black_fleet_budget.py:164) and the same "hosted keeps Black's native CLI" paragraph (docs/ci/ci-and-reviews.md:519). A git merge-tree of the two heads conflicts in all three files, so only one can land as written.

Only here: the .github/workflows/ci.yml:512-516 step comment. PR #13745 leaves "Hosted CI keeps the native command" stale, because a fork edit under .github/ trips the Fork Workflow-Change Guard, and its author deferred that line to a maintainer. This head also has readiness passed and all four lanes PASS.

Only in PR #13745:

  • scripts/bounded_black.py: memory_worker_budget() / hosted_worker_count(). These cap hosted --workers by MemAvailable (2 GiB per worker, one reserved for the coordinator) and log black hosted workers: N.
  • scripts/ci_black_diagnostics.py: a 30 s memory sampler around the gate (memory_cgroups, own_memory_current, start_sampler), with tests.
  • Fixes #13386, the open canonical issue. This PR says Fixes #13689, and issue CI: fork PR lint job killed mid-black on ubuntu-latest (exit 143) #13689 is already closed as a duplicate, so merging as written leaves issue CI: Backend Lint's black step killed by runner shutdown (exit 143) on 4 consecutive fork-PR heads #13386 open.
  • The only hosted proof: run 36110410422 passed on ubuntu-latest. The black step took 12m30s with black hosted workers: 4 (cpus=4), and the job took 21m19s of its 25-minute budget. On that 4-vCPU runner the cap comes out at 4, the same pool this PR gets from Black's default. So that run also covers the gap both Fable lanes named here: this PR's own CI runs on the fleet.

Suggestion: land this PR first, as the host. Before merge, change the closing line to Fixes #13386, keep issue #13689 as a reference, and cite the run above as hosted evidence. PR #13745 can then rebase down to its unique delta: the sampler and, if maintainers want it, the MemAvailable cap. That delta lives entirely in scripts/ and test/, which a fork can edit. If the cap is kept, the ci.yml comment needs one more clause ("hosted also sizes the pool to MemAvailable").

From a full open-PR audit against origin/main c66f95b (2026-09-25); findings were checked by independent reviewers reading the code. Reply here if this is wrong.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CI: fork PR lint job killed mid-black on ubuntu-latest (exit 143)

3 participants