Skip to content

Commit a44a3f9

Browse files
net0wellclaude
andcommitted
fix(core): limite de memória por cgroup v2 e threads por processo (spec 001)
Etapas com numpy morriam como INTERRUPTED em máquinas com muitos núcleos. No Linux o limite era RLIMIT_AS, que conta memória reservada, e o OpenBLAS reserva memória por thread: com 24 CPUs, só o `import numpy` passava do limite padrão de 1 GB (D-180). - Onde o host pode criar sub-cgroups (systemd com Delegate=yes, contêiner privilegiado como root), o runner usa cgroup v2: - move os próprios processos para psr-host e liga o controlador memory; - cada tentativa ganha psr-runs/<id> com memory.max, e o SDK entra nele antes de importar o script; - o OOM do kernel vira MemoryLimitExceeded (oom_kill em memory.events), a árvore é encerrada com cgroup.kill e o cgroup é removido. - Plano B (contêiner padrão, com o cgroup somente leitura): RLIMIT_AS e OMP/OPENBLAS/MKL/NUMEXPR/VECLIB limitados a ThreadsPerProcess. A opção é nova; nula, vale 1 no plano B e sem limite com cgroup. O modo usado é registrado uma vez. - Testes: unitários da montagem do cgroup sobre um sistema de arquivos falso; AC-29 (threads no plano B e valor configurado); pytest da entrada no cgroup e do plano B; AC-30 em contêiner privilegiado (OOM pelo kernel, numpy sem limite de threads). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 parent 039a8cc commit a44a3f9

15 files changed

Lines changed: 616 additions & 10 deletions

File tree

‎python/psrunner/src/psrunner/host.py‎

Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -17,8 +17,10 @@
1717
from psrunner.context import Context
1818
from psrunner.convert import inputs_to_python, outputs_to_wire
1919
from psrunner.fields import specs_from_dicts
20-
from psrunner.limits import apply_memory_limit
20+
from psrunner.limits import apply_limits
2121
from psrunner.protocol import (
22+
ENV_CGROUP,
23+
ENV_FALLBACK_THREADS,
2224
ENV_INPUT,
2325
ENV_MEMORY_LIMIT_MB,
2426
ENV_OUTPUT,
@@ -66,7 +68,11 @@ def _run() -> dict[str, Any]:
6668
f"this psrunner speaks {PROTOCOL_VERSION}."
6769
)
6870

69-
apply_memory_limit(int(os.environ.get(ENV_MEMORY_LIMIT_MB, "0") or 0))
71+
apply_limits(
72+
int(os.environ.get(ENV_MEMORY_LIMIT_MB, "0") or 0),
73+
os.environ.get(ENV_CGROUP, ""),
74+
int(os.environ.get(ENV_FALLBACK_THREADS, "0") or 0),
75+
)
7076
logger = _configure_logging()
7177

7278
try:

‎python/psrunner/src/psrunner/limits.py‎

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,12 +7,55 @@
77

88
from __future__ import annotations
99

10+
import os
1011
import sys
12+
from pathlib import Path
1113
from typing import Any
1214

1315
_MB = 1024 * 1024
1416
_keep_alive: list[Any] = []
1517

18+
# Thread pools of the numeric libraries (OpenMP, OpenBLAS, MKL, numexpr, Accelerate).
19+
THREAD_VARIABLES = (
20+
"OMP_NUM_THREADS",
21+
"OPENBLAS_NUM_THREADS",
22+
"MKL_NUM_THREADS",
23+
"NUMEXPR_NUM_THREADS",
24+
"VECLIB_MAXIMUM_THREADS",
25+
)
26+
27+
28+
def apply_limits(limit_mb: int, cgroup: str, fallback_threads: int) -> str:
29+
"""Applies the memory limit of the attempt; returns how: ``cgroup``, ``process`` or ``none``.
30+
31+
On Linux the runner may create a cgroup v2 with ``memory.max`` for the attempt (spec 001,
32+
D-180): joining it limits the memory really used. Without it, the limit falls back to
33+
``RLIMIT_AS``, which counts reserved address space, so the numeric libraries are limited to
34+
``fallback_threads`` threads: OpenBLAS reserves memory per thread and, on a machine with many
35+
cores, ``import numpy`` alone would exceed the limit. Both happen before the script is imported.
36+
"""
37+
if cgroup and join_cgroup(cgroup):
38+
return "cgroup"
39+
if fallback_threads > 0:
40+
limit_threads(fallback_threads)
41+
return "process" if apply_memory_limit(limit_mb) else "none"
42+
43+
44+
def join_cgroup(path: str) -> bool:
45+
"""Moves this process into the cgroup at ``path``. Returns ``True`` when it joined."""
46+
try:
47+
Path(path, "cgroup.procs").write_text(str(os.getpid()), encoding="ascii")
48+
except OSError as exc:
49+
print(f"psrunner: could not join the cgroup {path}: {exc}", file=sys.stderr)
50+
return False
51+
return True
52+
53+
54+
def limit_threads(count: int) -> None:
55+
"""Limits the thread pools of the numeric libraries, keeping values set by the runner."""
56+
for name in THREAD_VARIABLES:
57+
os.environ.setdefault(name, str(count))
58+
1659

1760
def apply_memory_limit(limit_mb: int) -> bool:
1861
"""Limits the memory of this process. Returns ``True`` when a limit was applied."""

‎python/psrunner/src/psrunner/protocol.py‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,10 @@
1010
ENV_WORK_DIR = "PSR_WORK_DIR"
1111
ENV_SCRIPT = "PSR_SCRIPT"
1212
ENV_MEMORY_LIMIT_MB = "PSR_MEMORY_LIMIT_MB"
13+
# Linux: cgroup v2 of the attempt (memory.max) and, if it cannot be joined, the threads of the
14+
# numeric libraries.
15+
ENV_CGROUP = "PSR_CGROUP"
16+
ENV_FALLBACK_THREADS = "PSR_FALLBACK_THREADS"
1317
ENV_SECRET_PREFIX = "PSR_SECRET_"
1418

1519
FILE_MARKER = "$file"

‎python/psrunner/tests/conftest.py‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -51,6 +51,7 @@ def _run(
5151
secrets: dict[str, str] | None = None,
5252
protocol: str = "1",
5353
memory_limit_mb: int = 0,
54+
extra_env: dict[str, str] | None = None,
5455
) -> HostRun:
5556
work = tmp_path / "work"
5657
for sub in ("in", "out", "tmp"):
@@ -110,6 +111,7 @@ def _run(
110111
)
111112
for name, value in (secrets or {}).items():
112113
env[f"PSR_SECRET_{name}"] = value
114+
env.update(extra_env or {})
113115

114116
completed = subprocess.run(
115117
[sys.executable, "-I", "-X", "utf8", "-m", "psrunner.host"],

‎python/psrunner/tests/test_host.py‎

Lines changed: 83 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,8 @@
22

33
from __future__ import annotations
44

5+
from pathlib import Path
6+
57
import pytest
68
from conftest import RunHost
79

@@ -267,6 +269,87 @@ def run(inputs, ctx):
267269
assert run.error["kind"] == "MemoryLimitExceeded"
268270

269271

272+
def test_spec001_ac30_joins_the_cgroup_of_the_attempt_instead_of_limiting_itself(
273+
run_host: RunHost, tmp_path: Path
274+
) -> None:
275+
# A stand-in for the cgroup the runner created: the real memory.max lives in the kernel.
276+
cgroup = tmp_path / "psr-runs" / "tentativa"
277+
cgroup.mkdir(parents=True)
278+
(cgroup / "cgroup.procs").write_text("", encoding="ascii")
279+
script = """
280+
import os
281+
from psrunner import stage
282+
283+
@stage
284+
def run(inputs, ctx):
285+
data = bytearray(256 * 1024 * 1024)
286+
return {"pid": os.getpid(), "threads": os.environ.get("OPENBLAS_NUM_THREADS", "")}
287+
"""
288+
289+
run = run_host(
290+
script,
291+
output_fields=[{"name": "pid", "type": "integer"}, {"name": "threads", "type": "string"}],
292+
memory_limit_mb=128,
293+
extra_env={"PSR_CGROUP": str(cgroup), "PSR_FALLBACK_THREADS": "1"},
294+
)
295+
296+
assert (cgroup / "cgroup.procs").read_text(encoding="ascii") == str(run.outputs["pid"])
297+
assert run.outputs["threads"] == "", "with a cgroup the numeric libraries keep their threads"
298+
299+
300+
def test_spec001_ac29_without_the_cgroup_limits_the_threads_and_keeps_the_memory_limit(
301+
run_host: RunHost, tmp_path: Path
302+
) -> None:
303+
script = """
304+
import os
305+
from psrunner import stage
306+
307+
@stage
308+
def run(inputs, ctx):
309+
names = [
310+
"OMP_NUM_THREADS",
311+
"OPENBLAS_NUM_THREADS",
312+
"MKL_NUM_THREADS",
313+
"NUMEXPR_NUM_THREADS",
314+
]
315+
threads = ",".join(os.environ.get(name, "") for name in names)
316+
data = bytearray(1024 * 1024 * 1024)
317+
return {"threads": threads, "tamanho": len(data)}
318+
"""
319+
320+
run = run_host(
321+
script,
322+
output_fields=[
323+
{"name": "threads", "type": "string"},
324+
{"name": "tamanho", "type": "integer"},
325+
],
326+
memory_limit_mb=128,
327+
extra_env={"PSR_CGROUP": str(tmp_path / "nao-existe"), "PSR_FALLBACK_THREADS": "3"},
328+
)
329+
330+
assert run.error["kind"] == "MemoryLimitExceeded"
331+
assert "could not join the cgroup" in run.stderr
332+
333+
334+
def test_spec001_ac29_fallback_threads_reach_the_script(run_host: RunHost, tmp_path: Path) -> None:
335+
script = """
336+
import os
337+
from psrunner import stage
338+
339+
@stage
340+
def run(inputs, ctx):
341+
return {"threads": os.environ.get("OPENBLAS_NUM_THREADS", "")}
342+
"""
343+
344+
run = run_host(
345+
script,
346+
output_fields=[{"name": "threads", "type": "string"}],
347+
extra_env={"PSR_CGROUP": str(tmp_path / "nao-existe"), "PSR_FALLBACK_THREADS": "3"},
348+
)
349+
350+
assert run.outputs["threads"] == "3"
351+
352+
270353
def test_spec002_ac11_protocol_mismatch(run_host: RunHost) -> None:
271354
run = run_host("from psrunner import stage\n", protocol="2")
272355

‎src/PythonScriptRunner.Core/DependencyInjection/PythonScriptRunnerOptionsValidator.cs‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -50,6 +50,11 @@ public ValidateOptionsResult Validate(string? name, PythonScriptRunnerOptions op
5050
errors.Add("MaxConcurrentProcesses must be at least 1.");
5151
}
5252

53+
if (options.ThreadsPerProcess is < 1)
54+
{
55+
errors.Add("ThreadsPerProcess must be at least 1 (or null).");
56+
}
57+
5358
if (string.IsNullOrWhiteSpace(options.PsrunnerRequirement))
5459
{
5560
errors.Add("PsrunnerRequirement is required.");

‎src/PythonScriptRunner.Core/DependencyInjection/PythonScriptRunnerServiceCollectionExtensions.cs‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -39,6 +39,7 @@ public static IPythonScriptRunnerBuilder AddPythonScriptRunner(
3939
services.TryAddSingleton<ISecretProvider, ConfigurationSecretProvider>();
4040
services.TryAddSingleton<IRunRecorder, NullRunRecorder>();
4141
services.TryAddSingleton<IEnvironmentManager, UvEnvironmentManager>();
42+
services.TryAddSingleton<CgroupMemory>();
4243
services.TryAddSingleton<IPythonExecutor, ProcessPythonExecutor>();
4344
services.TryAddSingleton<PsrMetrics>();
4445
services.TryAddSingleton<WorkspaceJanitor>();

‎src/PythonScriptRunner.Core/Diagnostics/PsrDiagnostics.cs‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -76,4 +76,10 @@ internal static partial class Log
7676

7777
[LoggerMessage(10, LogLevel.Error, "Could not record the failure of run {RunId}")]
7878
public static partial void RecordingFailed(ILogger logger, RunId runId, Exception exception);
79+
80+
[LoggerMessage(11, LogLevel.Information, "Memory limits of Python processes use cgroup v2 under {Path}")]
81+
public static partial void CgroupMemoryLimits(ILogger logger, string path);
82+
83+
[LoggerMessage(12, LogLevel.Information, "Memory limits of Python processes use RLIMIT_AS ({Reason}); numeric libraries get {Threads} thread(s) per process")]
84+
public static partial void ProcessMemoryLimits(ILogger logger, string reason, int threads);
7985
}

0 commit comments

Comments
 (0)