Skip to content

fix(ci): bound Black's workers on hosted runners (#13386) - #13745

Closed
jeeshofone wants to merge 2 commits into
kirodotdev:mainfrom
jeeshofone:fix/13386-bound-hosted-black
Closed

jeeshofone wants to merge 2 commits into
kirodotdev:mainfrom
jeeshofone:fix/13386-bound-hosted-black

Conversation

@jeeshofone

@jeeshofone jeeshofone commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

On GitHub-hosted fork runners (ubuntu-latest), Backend Lint & Type Check (3.12) dies in Check formatting (black, baselined) with exit 143 and "The runner has received a shutdown signal", 7-15 min into the step, on every fork PR head (#13386, with datapoints from #13109 and #7818). The gate never reaches a verdict, and Coverage Gate fails behind it.

Why it matters

Every fork PR's Backend Lint is red regardless of its diff, so no fork contribution can reach a green readiness rollup, and a new head does not clear it.

What changed (motivation → approach → change)

Symptom → cause: scripts/check_black_formatting.py exempted RUNNER_ENVIRONMENT=github-hosted from scripts/bounded_black.py, so hosted ran native python -m black with the default worker count (CPU count) and no recycling. Hosted runners set no job cgroup limit (memory.max: max; the pre-step snapshot on a killed run shows 14.9 GB of 16.4 GB free and ~6 GB already in system.slice), so the compiled-Black retention measured in #11880 grows until the VM, not a cgroup, runs out, which takes the runner service down with it. That matches the triage lead on the issue: oom_kill 0 on every visible cgroup, and no "after" snapshot because the step itself is killed.

Change, scripts/ only (no .github/ edit, so the Fork Workflow-Change Guard is not involved):

  • scripts/check_black_formatting.py: every runner now goes through bounded_black.py (one file per spawned worker, 2 GiB RLIMIT_AS per process on Linux). On github-hosted it also passes --workers N and prints black hosted workers: N.
  • scripts/bounded_black.py: hosted_worker_count() = min(CPUs, MAX_WORKERS=8, memory_worker_budget()), where the budget is the number of 2 GiB ceilings that fit in MemAvailable after reserving one for the coordinator. So the per-process caps of every live process together fit in the memory the VM had free at start. Unreadable meminfo falls back to 2, never to the CPU count. On ubuntu-latest (4 vCPU, ~14.9 GB free) that is 4 workers.
  • scripts/ci_black_diagnostics.py: a daemon thread prints a flushed black memory sample t=<s>s MemAvailable=... SwapFree=... cgroup_memory_current=... line every 30 s while the gate runs and stops when the gate returns. If a runner is still killed, the last line before death records memory. The cgroup reading reuses the snapshot's own cgroup walk (factored into memory_cgroups(), with byte-identical snapshot output), taking the nearest readable memory.current from the process's leaf upward. Sampler errors, including a sampler thread that cannot start, print only the exception type; the gate still runs once and its verdict is unchanged.
  • Fleet/local worker selection (BLACK_NUM_WORKERS or CPUs, capped at 8) is unchanged.
  • docs/ci/ci-and-reviews.md describes the new hosted path and the sampler.

Alternatives not taken: capping hosted at 2 workers. Measured locally, bounded Black costs ~35% more CPU than native (spawn per file: 39.6 vs 29.2 CPU-min for the whole tree), so 2 workers on a 4 vCPU hosted VM risks the job's 25 min timeout-minutes. The memory-sized bound keeps all four CPUs while guaranteeing the caps fit.

Follow-up for a maintainer (out of this PR's reach, as a fork cannot edit workflows): the step comment at .github/workflows/ci.yml:512-514 still says hosted keeps the native command and default worker count.

Backwards compatibility

Compatible: the gate's inputs, verdicts, exit codes, ratchet scope and baseline handling are unchanged. Only the hosted launcher and its worker count change, and the diagnostic adds log lines.

Tests

test/test_black_fleet_budget.py:

  • test_memory_budget_fits_every_address_space_ceiling: pins the budget (14.9 GB free → 6; 3 GB → 1; malformed/missing/non-kB meminfo → None) and that (budget + 1) × 2 GiB ≤ MemAvailable.
  • test_hosted_worker_count_is_bounded: 4 CPUs → 4; 64 CPUs → 6 (memory-bound); 64 GiB free → 8 (launcher ceiling); unreadable meminfo → 2; unknown CPU count → 1.
  • test_hosted_gate_never_runs_native_black: a hosted gate run calls bounded_black.py --workers 6, never -m black.
  • test_gate_preserves_scope_workers_and_baseline_on_black_result (updated): hosted argv is now the bounded launcher plus --workers.
  • test_sampler_records_memory_while_the_gate_runs: a real subprocess run prints ≥2 samples during a 1.5 s gate with real MemAvailable, and none after the verdict line.
  • test_sampler_that_cannot_start_still_runs_the_gate, test_sampler_failure_cannot_change_the_verdict, test_own_memory_current_reads_the_process_cgroup (leaf, and nearest-ancestor fallback when the leaf hides memory.current).

All new/updated tests fail on origin/main's scripts (32 failures) and pass with this change. pytest test/test_black_fleet_budget.py test/test_black_gate_contract.py test/test_ci_nonroot_boundary.py: 125 passed.

Manual verification

  • Full gate locally as hosted: RUNNER_ENVIRONMENT=github-hosted python3 scripts/ci_black_diagnostics.py scripts/check_black_formatting.py passed ("black gate passed ... 967 known-unformatted file(s) still listed") in ~5.5 min with a cold cache, printing 11 memory samples (cgroup memory.current steady at 2.8-3.0 GB).
  • black (hunk-clean on touched files), isort, flake8 and per-file mypy on the changed scripts; comment-history, subprocess-encoding, sync-io-in-async, brand, harness, feature-map, changelog, testpaths and docs-lint gates pass.
  • Hosted proof on the first head (75de72b): Backend Lint & Type Check (3.12) on ubuntu-latest passed; the black step ran 12m30s with black hosted workers: 4 (cpus=4) and MemAvailable flat at 14.9-15.2 GB across every 30 s sample; job total 21m19s against the 25 min budget.
  • Second head (adc8d59): Backend Lint & Type Check (3.12) passed again on ubuntu-latest (21m30s).
  • The proof that matters is this PR's own hosted Backend Lint & Type Check (3.12) run finishing the black step with a verdict rather than exit 143.

Related Issues

Fixes #13386

Pattern harvest

Not generalizable: the defect is one environment-specific exemption from an existing bound; the class (unbounded parallel tool runs on hosted runners) has one member in this repo.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

…ring the gate (kirodotdev#13386)

Hosted fork runners ran the black gate as native `python -m black` with the
default worker count and no job cgroup limit; the walk exhausted the VM and the
runner service died mid-step with exit 143 on every fork head.

- check_black_formatting.py routes every runner through bounded_black.py
  (one file per spawned worker, 2 GiB address-space ceiling per process) and,
  on github-hosted, passes --workers = min(CPUs, 8, 2 GiB ceilings that fit in
  MemAvailable after reserving one for the coordinator); unreadable meminfo
  falls back to two workers.
- ci_black_diagnostics.py prints a flushed memory sample (MemAvailable,
  SwapFree, own cgroup memory.current) every 30 s while the gate runs, so a
  killed runner still leaves its last reading.
- Fleet/local worker selection is unchanged. No workflow file changes.
@jeeshofone
jeeshofone requested review from a team and buluoray September 25, 2026 07:58
@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: action required A blocking check or review needs attention labels Sep 25, 2026
@github-actions

github-actions Bot commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5, fork) — ✅ PASS

Design-level review of 562be441fb92eb165f321f63af1b57a20afd5d8f via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Merge-time follow-up: .github/workflows/ci.yml:512-514 still documents the hosted native-Black path this removes; a maintainer must fix it, forks cannot.

[DESIGN-REVIEWED] 562be44

@github-actions

github-actions Bot commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review (fork) — ✅ no blocking findings

Reviewed 562be441fb92eb165f321f63af1b57a20afd5d8f via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 562be44

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 25, 2026
@github-actions

github-actions Bot commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5, fork) — ✅ PASS

Premise-level review of 562be441fb92eb165f321f63af1b57a20afd5d8f via the fork AI-review pipeline — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

Verify this PR's own hosted Backend Lint finishes inside the 25-minute budget — both proof heads landed at 21m19s–21m30s, leaving under 4 minutes of headroom.

What this change ships

Inventory (7 items) — 7 justified

Intent: make fork-PR Backend Lint stop dying with exit 143 by bounding Black's memory on GitHub-hosted runners — a FIX (issue #13386, two linked hosted proof runs).

  1. Hosted runners now run the black gate through the recycling, 2 GiB-capped launcher instead of native black — justified
  2. Hosted worker count is sized to free memory at start (min of CPUs, 8, memory budget) and logged — justified
  3. Unreadable meminfo falls back to 2 workers, never the CPU count — justified
  4. A black memory sample log line prints every 30 s during the gate — rides along, declared; harm derived (a killed runner leaves no "after" snapshot, per the issue's triage)
  5. Sampler failure prints only the exception type and cannot change the verdict — justified
  6. Snapshot's cgroup walk factored into memory_cgroups(), output unchanged — rides along, internal, no surface
  7. docs/ci/ci-and-reviews.md hosted-path section rewritten in the same commit — justified

The reversed pins (the hosted-native comment, doc sentence, and -m black test assertion) are declared as the cause, with a linked issue and two passing hosted runs as evidence — lens 8 satisfied. --workers reuses Black's native CLI option through the base launcher's patched executor; no new flag or config surface ships. Grepped RUNNER_ENVIRONMENT: the black gate was the only hosted exemption, so zero unfixed siblings, matching the description's "one member" claim.

[FIRST-PRINCIPLES-REVIEWED] 562be44

@github-actions

github-actions Bot commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5 Review (fork) — ✅ no blocking findings

Reviewed 562be441fb92eb165f321f63af1b57a20afd5d8f via the fork AI-review pipeline; updated in place on each push.

Review details

Hosted CI's step comment now asserts the exact path this change deletes.

FINDING — .github/workflows/ci.yml:513 — the comment directly above the changed step still says Hosted CI keeps the native command and default worker count, which this diff makes false (hosted now runs bounded_black.py with an explicit --workers), so the next person debugging a hosted formatter failure reasons from a launcher that no longer exists → Fix: replace that sentence in this commit with "every runner runs the gate through scripts/bounded_black.py; hosted additionally sizes the pool to MemAvailable at start".

[OPUS-REVIEWED] 562be44

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 25, 2026
@jeeshofone

Copy link
Copy Markdown
Contributor Author

Fixed in adc8d59 (span 36451860f461). scripts/ci_black_diagnostics.py now carries one cgroup walk: the snapshot's mountinfo//proc/self/cgroup resolution is factored into memory_cgroups() (snapshot output byte-identical on a real host, checked by diffing the probe paths before/after), and own_memory_current() iterates that walk for the nearest readable cgroup v2 memory.current. start_sampler() takes no parameter; the period is SAMPLE_SECONDS, which the tests override at module level.

@jeeshofone

Copy link
Copy Markdown
Contributor Author

Accepted, deferred to a maintainer (span ff43b6f1b705). The comment at .github/workflows/ci.yml:512-514 is now stale: it says hosted keeps the native command and default worker count, and after this change every runner goes through scripts/bounded_black.py with hosted also passing --workers. This PR is from a fork, and a fork edit under .github/ trips the Fork Workflow-Change Guard, so the fix stays in scripts/ on purpose. Suggested replacement for a maintainer follow-up: "Every runner recycles workers capped at 2 GiB each on Linux; hosted runners also size the pool to MemAvailable (scripts/bounded_black.py). Keep cgroup evidence: per-process limits do not bound the whole job." The PR body records this follow-up and docs/ci/ci-and-reviews.md already describes the new behaviour.

@jeeshofone

Copy link
Copy Markdown
Contributor Author

Fixed in adc8d59 (span f48d0288e21f). test_memory_budget_fits_every_address_space_ceiling now selects the MemAvailable: line before parsing, so on HOSTED_MEMINFO it checks (6 + 1) x 2 GiB <= 14,950,508 kB, not against MemTotal.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 25, 2026
@bolichen97

Copy link
Copy Markdown
Contributor

Consolidation: this PR and PR #13698 ship the same core fix for issue #13386

Overlap. Both PRs delete the RUNNER_ENVIRONMENT == "github-hosted" native -m black branch in scripts/check_black_formatting.py::_unformatted and replace it with the same line:

launcher = [str(Path(__file__).with_name("bounded_black.py"))]

That line is scripts/check_black_formatting.py:114 here and line 100 in PR #13698. Both also rewrite the hosted-launcher expectation in test_gate_preserves_scope_workers_and_baseline_on_black_result and the same "hosted keeps Black's native CLI" paragraph in docs/ci/ci-and-reviews.md. Each PR merges cleanly into main by itself, but whichever lands second will conflict in all three files.

Only in this PR:

  • memory_worker_budget() and hosted_worker_count() (scripts/bounded_black.py:21, :43), plus the hosted --workers N append and log line.
  • The 30 s memory sampler in scripts/ci_black_diagnostics.py (memory_cgroups, own_memory_current, sample, start_sampler), and the tests for the cap and the sampler.
  • Fixes #13386, the open issue.
  • The only hosted proof run (actions run 36110410422).

Only in PR #13698: the rewrite of the step comment at .github/workflows/ci.yml:512-513. This is the follow-up you deferred here because a fork can't edit .github/. PR #13698 currently says Fixes #13689, and issue #13689 is already closed as a duplicate.

Same behaviour today. On ubuntu-latest (4 vCPU, about 14.9 GB MemAvailable) both PRs run 4 workers:

So your hosted run is also evidence for PR #13698's configuration. The memory cap is defensive: it only changes the count when MemAvailable is below (cpus + 1) x 2 GiB.

Suggested sequencing:

  1. Let PR ci(lint): run hosted black through bounded_black #13698 carry the core fix. It is earlier, same-repo, readiness passed, and it already fixes the ci.yml comment. Before it merges it should add Fixes #13386 and could cite run 36110410422 as hosted evidence.
  2. Once it lands, rebase this PR down to what only it has, all in scripts/ and test/, which a fork can edit:
    • the sampler and its three tests;
    • optionally the hosted cap, with _read_text, _load_bounded_black and their tests. If the cap stays, a maintainer should add one clause to the ci.yml comment: hosted also sizes the pool to MemAvailable.
  3. The current readiness blocker, GPT F1 on the unguarded thread.start() at scripts/ci_black_diagnostics.py:162, is in the sampler, so it comes along with the rebase. Guarding start_sampler() the same way diagnose() guards snapshot() would clear it.

From a full open-PR audit against origin/main c66f95b (2026-09-25); findings were checked by independent reviewers reading the code. Reply here if this is wrong.

…irodotdev#13386)

own_memory_current() now reads the nearest readable cgroup v2
memory.current via the snapshot's memory_cgroups() walk instead of a
second resolver, start_sampler() takes no unused parameter, a sampler that
cannot start prints its exception type and the gate still runs, and the
budget test reads MemAvailable rather than MemTotal from the fixture.
@jeeshofone
jeeshofone force-pushed the fix/13386-bound-hosted-black branch from adc8d59 to 562be44 Compare September 25, 2026 10:59
@jeeshofone

Copy link
Copy Markdown
Contributor Author

Fixed in 562be44 (span 240e20c5e9ff). main() in scripts/ci_black_diagnostics.py now starts the sampler inside its own try: if start_sampler() raises (e.g. RuntimeError: can't start new thread), it prints black memory sampler unavailable: <ExceptionType> and the gate still runs exactly once with its verdict unchanged; the stop/join in finally only runs when a sampler exists. Regression: test_sampler_that_cannot_start_still_runs_the_gate patches threading.Thread.start to raise, asserts one gate call, returncode preserved, and no exception payload printed. It fails on adc8d59 and passes on this head.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 25, 2026
@github-actions github-actions Bot added the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Sep 25, 2026
@jeeshofone

Copy link
Copy Markdown
Contributor Author

Closing as superseded by #13698, which merged at 19:17 UTC on 25 Sep and runs hosted black through scripts/bounded_black.py.

Since then the kill has not come back. Of the 40 fork-PR CI runs created after 19:30 UTC on 25 Sep, 29 Backend Lint jobs passed and none died with exit 143. The one failure (run 36237060823) reported a real unformatted file in that PR, with oom_kill 0.

This branch also sized hosted workers from MemAvailable and printed a memory sample every 30 seconds while the gate ran. Main does not need either today. If the exit-143 kill returns, fix/13386-r2 at 562be441f still has both.

@jeeshofone jeeshofone closed this Sep 26, 2026
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Sep 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor) merge conflict Branch has merge conflicts with its base — author must resolve before merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CI: Backend Lint's black step killed by runner shutdown (exit 143) on 4 consecutive fork-PR heads

2 participants