Repository navigation
fix(ci): bound Black's workers on hosted runners (#13386) - #13745
jeeshofone wants to merge 2 commits into
Conversation
…ring the gate (kirodotdev#13386) Hosted fork runners ran the black gate as native `python -m black` with the default worker count and no job cgroup limit; the walk exhausted the VM and the runner service died mid-step with exit 143 on every fork head. - check_black_formatting.py routes every runner through bounded_black.py (one file per spawned worker, 2 GiB address-space ceiling per process) and, on github-hosted, passes --workers = min(CPUs, 8, 2 GiB ceilings that fit in MemAvailable after reserving one for the coordinator); unreadable meminfo falls back to two workers. - ci_black_diagnostics.py prints a flushed memory sample (MemAvailable, SwapFree, own cgroup memory.current) every 30 s while the gate runs, so a killed runner still leaves its last reading. - Fleet/local worker selection is unchanged. No workflow file changes.
Design Review (Fable 5, fork) — ✅ PASSDesign-level review of Design-Verdict: PASS Merge-time follow-up: [DESIGN-REVIEWED] 562be44 |
GPT 5.6 Review (fork) — ✅ no blocking findingsReviewed Review detailsNo findings. |
First Principles Review (Fable 5, fork) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Verify this PR's own hosted Backend Lint finishes inside the 25-minute budget — both proof heads landed at 21m19s–21m30s, leaving under 4 minutes of headroom. What this change shipsInventory (7 items) — 7 justifiedIntent: make fork-PR Backend Lint stop dying with exit 143 by bounding Black's memory on GitHub-hosted runners — a FIX (issue #13386, two linked hosted proof runs).
The reversed pins (the hosted-native comment, doc sentence, and [FIRST-PRINCIPLES-REVIEWED] 562be44 |
Opus 5 Review (fork) — ✅ no blocking findingsReviewed Review detailsHosted CI's step comment now asserts the exact path this change deletes. FINDING — .github/workflows/ci.yml:513 — the comment directly above the changed step still says [OPUS-REVIEWED] 562be44 |
|
Fixed in adc8d59 (span 36451860f461). |
|
Accepted, deferred to a maintainer (span ff43b6f1b705). The comment at |
|
Fixed in adc8d59 (span f48d0288e21f). |
|
Consolidation: this PR and PR #13698 ship the same core fix for issue #13386 Overlap. Both PRs delete the launcher = [str(Path(__file__).with_name("bounded_black.py"))]That line is Only in this PR:
Only in PR #13698: the rewrite of the step comment at Same behaviour today. On
So your hosted run is also evidence for PR #13698's configuration. The memory cap is defensive: it only changes the count when MemAvailable is below (cpus + 1) x 2 GiB. Suggested sequencing:
From a full open-PR audit against origin/main c66f95b (2026-09-25); findings were checked by independent reviewers reading the code. Reply here if this is wrong. |
…irodotdev#13386) own_memory_current() now reads the nearest readable cgroup v2 memory.current via the snapshot's memory_cgroups() walk instead of a second resolver, start_sampler() takes no unused parameter, a sampler that cannot start prints its exception type and the gate still runs, and the budget test reads MemAvailable rather than MemTotal from the fixture.
adc8d59 to
562be44
Compare
|
Fixed in 562be44 (span 240e20c5e9ff). |
|
Closing as superseded by #13698, which merged at 19:17 UTC on 25 Sep and runs hosted black through Since then the kill has not come back. Of the 40 fork-PR CI runs created after 19:30 UTC on 25 Sep, 29 Backend Lint jobs passed and none died with exit 143. The one failure (run 36237060823) reported a real unformatted file in that PR, with This branch also sized hosted workers from |
Problem / Motivation
On GitHub-hosted fork runners (
ubuntu-latest),Backend Lint & Type Check (3.12)dies inCheck formatting (black, baselined)with exit 143 and "The runner has received a shutdown signal", 7-15 min into the step, on every fork PR head (#13386, with datapoints from #13109 and #7818). The gate never reaches a verdict, and Coverage Gate fails behind it.Why it matters
Every fork PR's Backend Lint is red regardless of its diff, so no fork contribution can reach a green readiness rollup, and a new head does not clear it.
What changed (motivation → approach → change)
Symptom → cause:
scripts/check_black_formatting.pyexemptedRUNNER_ENVIRONMENT=github-hostedfromscripts/bounded_black.py, so hosted ran nativepython -m blackwith the default worker count (CPU count) and no recycling. Hosted runners set no job cgroup limit (memory.max: max; the pre-step snapshot on a killed run shows 14.9 GB of 16.4 GB free and ~6 GB already insystem.slice), so the compiled-Black retention measured in #11880 grows until the VM, not a cgroup, runs out, which takes the runner service down with it. That matches the triage lead on the issue:oom_kill 0on every visible cgroup, and no "after" snapshot because the step itself is killed.Change,
scripts/only (no.github/edit, so the Fork Workflow-Change Guard is not involved):scripts/check_black_formatting.py: every runner now goes throughbounded_black.py(one file per spawned worker, 2 GiBRLIMIT_ASper process on Linux). Ongithub-hostedit also passes--workers Nand printsblack hosted workers: N.scripts/bounded_black.py:hosted_worker_count()= min(CPUs,MAX_WORKERS=8,memory_worker_budget()), where the budget is the number of 2 GiB ceilings that fit inMemAvailableafter reserving one for the coordinator. So the per-process caps of every live process together fit in the memory the VM had free at start. Unreadable meminfo falls back to 2, never to the CPU count. Onubuntu-latest(4 vCPU, ~14.9 GB free) that is 4 workers.scripts/ci_black_diagnostics.py: a daemon thread prints a flushedblack memory sample t=<s>s MemAvailable=... SwapFree=... cgroup_memory_current=...line every 30 s while the gate runs and stops when the gate returns. If a runner is still killed, the last line before death records memory. The cgroup reading reuses the snapshot's own cgroup walk (factored intomemory_cgroups(), with byte-identical snapshot output), taking the nearest readablememory.currentfrom the process's leaf upward. Sampler errors, including a sampler thread that cannot start, print only the exception type; the gate still runs once and its verdict is unchanged.BLACK_NUM_WORKERSor CPUs, capped at 8) is unchanged.docs/ci/ci-and-reviews.mddescribes the new hosted path and the sampler.Alternatives not taken: capping hosted at 2 workers. Measured locally, bounded Black costs ~35% more CPU than native (spawn per file: 39.6 vs 29.2 CPU-min for the whole tree), so 2 workers on a 4 vCPU hosted VM risks the job's 25 min
timeout-minutes. The memory-sized bound keeps all four CPUs while guaranteeing the caps fit.Follow-up for a maintainer (out of this PR's reach, as a fork cannot edit workflows): the step comment at
.github/workflows/ci.yml:512-514still says hosted keeps the native command and default worker count.Backwards compatibility
Compatible: the gate's inputs, verdicts, exit codes, ratchet scope and baseline handling are unchanged. Only the hosted launcher and its worker count change, and the diagnostic adds log lines.
Tests
test/test_black_fleet_budget.py:test_memory_budget_fits_every_address_space_ceiling: pins the budget (14.9 GB free → 6; 3 GB → 1; malformed/missing/non-kB meminfo →None) and that (budget + 1) × 2 GiB ≤MemAvailable.test_hosted_worker_count_is_bounded: 4 CPUs → 4; 64 CPUs → 6 (memory-bound); 64 GiB free → 8 (launcher ceiling); unreadable meminfo → 2; unknown CPU count → 1.test_hosted_gate_never_runs_native_black: a hosted gate run callsbounded_black.py --workers 6, never-m black.test_gate_preserves_scope_workers_and_baseline_on_black_result(updated): hosted argv is now the bounded launcher plus--workers.test_sampler_records_memory_while_the_gate_runs: a real subprocess run prints ≥2 samples during a 1.5 s gate with realMemAvailable, and none after the verdict line.test_sampler_that_cannot_start_still_runs_the_gate,test_sampler_failure_cannot_change_the_verdict,test_own_memory_current_reads_the_process_cgroup(leaf, and nearest-ancestor fallback when the leaf hidesmemory.current).All new/updated tests fail on
origin/main's scripts (32 failures) and pass with this change.pytest test/test_black_fleet_budget.py test/test_black_gate_contract.py test/test_ci_nonroot_boundary.py: 125 passed.Manual verification
RUNNER_ENVIRONMENT=github-hosted python3 scripts/ci_black_diagnostics.py scripts/check_black_formatting.pypassed ("black gate passed ... 967 known-unformatted file(s) still listed") in ~5.5 min with a cold cache, printing 11 memory samples (cgroupmemory.currentsteady at 2.8-3.0 GB).ubuntu-latestpassed; the black step ran 12m30s withblack hosted workers: 4 (cpus=4)andMemAvailableflat at 14.9-15.2 GB across every 30 s sample; job total 21m19s against the 25 min budget.ubuntu-latest(21m30s).Backend Lint & Type Check (3.12)run finishing the black step with a verdict rather than exit 143.Related Issues
Fixes #13386
Pattern harvest
Not generalizable: the defect is one environment-specific exemption from an existing bound; the class (unbounded parallel tool runs on hosted runners) has one member in this repo.
Checklist