diff --git a/docs/api/cli.md b/docs/api/cli.md index acf23d9..d6c7f5a 100644 --- a/docs/api/cli.md +++ b/docs/api/cli.md @@ -27,6 +27,10 @@ Run a Python script on a Globus Compute endpoint. Use -- to pass arguments to parameterized harnesses: hog run script.py harness -- arg1 --option=value +If the current environment does not satisfy the script's PEP 723 +requires-python / dependencies, hog run re-executes itself inside a uv +environment built from that metadata (disable with --no-bootstrap). + **Usage**: ```console @@ -42,6 +46,7 @@ $ hog run [OPTIONS] SCRIPT [HARNESS] * `--no-fun-allowed`: Suppress emoji output * `--log-level TEXT`: Set logging level (DEBUG, INFO, WARNING, ERROR) +* `--no-bootstrap`: Run the harness in the current environment even if it does not satisfy the script's requires-python / dependencies * `--help`: Show this message and exit. ## `hog init` diff --git a/docs/api/environment_variables.md b/docs/api/environment_variables.md index 5b478bf..0f7dd3b 100644 --- a/docs/api/environment_variables.md +++ b/docs/api/environment_variables.md @@ -84,6 +84,27 @@ export GROUNDHOG_CACHE_DIR=/gpfs/shared/uv-cache 3. `$TMPDIR` (temporary directory) 4. `/tmp` (system temp) +## GROUNDHOG_NO_BOOTSTRAP + +**Type:** boolean (any truthy value) + +**Default:** not set + +Keeps `hog run` in the current environment even when that environment does not satisfy the script's PEP 723 `requires-python` / `dependencies`. Normally `hog run` re-executes itself inside a uv environment built from the script metadata in that case (see [Functions and Harnesses](../concepts/functions-and-harnesses.md#the-driver-environment)). Equivalent to the `--no-bootstrap` flag. + +**Example:** +```bash +GROUNDHOG_NO_BOOTSTRAP=1 hog run script.py +``` + +## GROUNDHOG_BOOTSTRAPPED + +**Type:** boolean (any truthy value) + +**Default:** not set + +Set automatically by `hog run` in the environment of the child process it bootstraps. A child that still finds the environment unsuitable warns and runs in-process instead of bootstrapping again. You should not need to set this yourself. + ## `uv` Environment Variables Groundhog uses `uv` to manage Python environments on remote endpoints. Any `UV_*` environment variable can be used to override `[tool.uv]` configuration in your script. diff --git a/docs/concepts/functions-and-harnesses.md b/docs/concepts/functions-and-harnesses.md index b0bebd1..6bbdfec 100644 --- a/docs/concepts/functions-and-harnesses.md +++ b/docs/concepts/functions-and-harnesses.md @@ -47,6 +47,20 @@ hog run script.py # Runs the 'main' harness hog run script.py my_harness # Runs a specific harness ``` +### The driver environment + +`hog run` imports your script in the process it runs in, so the harness (and anything defined at module level, such as classes that subclass a third-party base) needs its imports available *locally* as well as on the endpoint. The PEP 723 header already lists them, so `hog run` uses it: when the current interpreter does not satisfy `requires-python`, or any entry in `dependencies` is not installed, `hog run` re-executes itself inside a uv environment built from that metadata: + +```bash +uv run --no-project --python "" \ + --with groundhog-hpc== --with ... \ + hog run script.py harness -- args +``` + +The `[tool.uv]` settings `exclude-newer`, `index-url` and `extra-index-url` are forwarded when the script sets them, so the driver resolves the same way the endpoint does. If the current environment already satisfies the header, nothing changes and the harness runs in-process. + +To keep the old in-process behavior, pass `--no-bootstrap` (before the script path) or set `GROUNDHOG_NO_BOOTSTRAP=1`; `hog run` then warns about the mismatch and continues. + ### Parameterized harnesses Harnesses can accept parameters that map to CLI arguments. This makes harnesses reusable without editing code: diff --git a/docs/examples/dependencies.md b/docs/examples/dependencies.md index 79b4fb1..d66ffa8 100644 --- a/docs/examples/dependencies.md +++ b/docs/examples/dependencies.md @@ -41,7 +41,7 @@ def main(): 1. Dependencies are declared in the PEP 723 metadata block. These will be installed automatically by uv when the function runs on the remote endpoint. See also: [uv script dependencies](https://docs.astral.sh/uv/guides/scripts/#declaring-script-dependencies) -2. Import remote dependencies **inside** the function, not at module level. This ensures imports happen after uv installs the packages in the remote environment. +2. Import dependencies that only the remote side needs **inside** the function. The environment `hog run` starts from need not have them installed, and importing lazily keeps startup fast. ## Adding Dependencies @@ -76,7 +76,9 @@ def process_data(data: list[float]) -> float: return np.std(data) ``` -**Why?** Module-level imports fail when the script loads locally (before uv installs packages on the remote endpoint). Function-level imports ensure the import happens after uv sets up the remote environment. +**Why?** A module-level import runs wherever the script is loaded, including on your laptop when `hog run` imports it. If a package is missing there, `hog run` will bootstrap a driver environment from the PEP 723 header (see [the driver environment](../concepts/functions-and-harnesses.md#the-driver-environment)), which works but costs a resolve on every run. Importing inside the function keeps the driver light and lets the harness run in-process whenever it can. + +Module-level imports are the right call when the harness itself needs the package, for example a class defined in the script that subclasses a third-party base. Declare the package in `dependencies` and `hog run` takes care of the driver side. ## Running the Example diff --git a/src/groundhog_hpc/app/run.py b/src/groundhog_hpc/app/run.py index a411085..989cc5f 100644 --- a/src/groundhog_hpc/app/run.py +++ b/src/groundhog_hpc/app/run.py @@ -2,6 +2,7 @@ import inspect import os +import subprocess import sys from pathlib import Path from typing import Any @@ -9,15 +10,15 @@ import typer from groundhog_hpc.app.utils import ( + bootstrap_reasons, + build_bootstrap_command, check_and_update_metadata, - python_version_matches, ) from groundhog_hpc.configuration.pep723 import read_pep723 from groundhog_hpc.errors import RemoteExecutionError from groundhog_hpc.harness import Harness from groundhog_hpc.logging import setup_logging from groundhog_hpc.utils import ( - get_groundhog_version_spec, import_user_script, path_to_module_name, ) @@ -71,11 +72,24 @@ def run( "--log-level", help="Set logging level (DEBUG, INFO, WARNING, ERROR)\n\n[env: GROUNDHOG_LOG_LEVEL=]", ), + no_bootstrap: bool = typer.Option( + False, + "--no-bootstrap", + help=( + "Run the harness in the current environment even if it does not " + "satisfy the script's requires-python / dependencies" + "\n\n[env: GROUNDHOG_NO_BOOTSTRAP=]" + ), + ), ) -> None: """Run a Python script on a Globus Compute endpoint. Use -- to pass arguments to parameterized harnesses: hog run script.py harness -- arg1 --option=value + + If the current environment does not satisfy the script's PEP 723 + requires-python / dependencies, hog run re-executes itself inside a uv + environment built from that metadata (disable with --no-bootstrap). """ # Handle the -- separator for harness arguments # ctx.args may contain ['--', 'arg1', 'arg2'] - strip the '--' if present @@ -108,28 +122,33 @@ def run( contents = check_and_update_metadata(script_path, contents) metadata = read_pep723(contents) - if metadata and metadata.requires_python: - requires_python = metadata.requires_python - current_version = f"{sys.version_info.major}.{sys.version_info.minor}.{sys.version_info.micro}" - - if not python_version_matches(current_version, requires_python): - groundhog_spec = get_groundhog_version_spec() - uv_cmd = ( - f"uv run --with {groundhog_spec} " - f"--python {requires_python} " - f"hog run {script_path} {harness}" - ) - + if metadata is not None and (reasons := bootstrap_reasons(metadata)): + skip_bootstrap = ( + no_bootstrap + or bool(os.environ.get("GROUNDHOG_NO_BOOTSTRAP")) + or bool(os.environ.get("GROUNDHOG_BOOTSTRAPPED")) + ) + if not skip_bootstrap: + # Re-run this same invocation inside a uv environment built from the + # script's metadata; the child sees GROUNDHOG_BOOTSTRAPPED and runs + # in-process, so a child that still fails the checks cannot loop. + cmd = build_bootstrap_command(script_path, harness, harness_args, metadata) typer.echo( - f"Warning: Script requires Python {requires_python}, " - f"but current version is {current_version}. This may " - "cause issues with serialization.", + f"Bootstrapping an environment from {script_path.name}'s " + f"metadata ({'; '.join(reasons)})", err=True, ) - typer.echo( - f"\nTo run with matching Python version, use:\n {uv_cmd}", - err=True, + completed = subprocess.run( + cmd, env={**os.environ, "GROUNDHOG_BOOTSTRAPPED": "1"} ) + raise typer.Exit(completed.returncode) + + typer.echo( + "Warning: running in the current environment although " + f"{'; '.join(reasons)}. Imports may fail and remote results may " + "not deserialize.", + err=True, + ) try: module_name = path_to_module_name(script_path) diff --git a/src/groundhog_hpc/app/utils.py b/src/groundhog_hpc/app/utils.py index 4d02cfe..df81434 100644 --- a/src/groundhog_hpc/app/utils.py +++ b/src/groundhog_hpc/app/utils.py @@ -1,20 +1,25 @@ """Shared utility functions for the Groundhog CLI.""" +import importlib.metadata import subprocess +import sys import tempfile from pathlib import Path import typer import uv +from packaging.requirements import InvalidRequirement, Requirement from packaging.specifiers import InvalidSpecifier, SpecifierSet from packaging.version import Version +import groundhog_hpc from groundhog_hpc.configuration.pep723 import ( Pep723Metadata, insert_or_update_metadata, read_pep723, write_pep723, ) +from groundhog_hpc.utils import get_groundhog_version_spec def normalize_python_version_with_uv(python: str) -> str: @@ -80,6 +85,127 @@ def python_version_matches(current: str, spec: str) -> bool: return Version(current) in SpecifierSet(spec) +def current_python_version() -> str: + """Return the running interpreter's version as "X.Y.Z".""" + v = sys.version_info + return f"{v.major}.{v.minor}.{v.micro}" + + +def missing_dependencies(dependencies: list[str]) -> list[str]: + """Return the entries of ``dependencies`` the current environment does not satisfy. + + An entry is satisfied when a distribution with its name is installed and, + unless the entry points at a URL, the installed version is within its + specifier. Entries whose environment marker evaluates false are skipped. + Malformed entries are reported as missing so that uv gets to raise the + real error when the environment is built. + + Args: + dependencies: PEP 508 requirement strings, e.g. from PEP 723 metadata + + Returns: + The unsatisfied entries, verbatim, in their original order + """ + missing: list[str] = [] + for dep in dependencies: + try: + req = Requirement(dep) + except InvalidRequirement: + missing.append(dep) + continue + if req.marker is not None and not req.marker.evaluate(): + continue + try: + installed = importlib.metadata.version(req.name) + except importlib.metadata.PackageNotFoundError: + missing.append(dep) + continue + if req.url is None and not req.specifier.contains(installed, prereleases=True): + missing.append(dep) + return missing + + +def bootstrap_reasons(metadata: Pep723Metadata) -> list[str]: + """Explain why the current environment cannot host the script's harness. + + Args: + metadata: The script's PEP 723 metadata + + Returns: + Human-readable reasons; empty when the environment satisfies both + ``requires-python`` and ``dependencies`` + """ + reasons: list[str] = [] + current = current_python_version() + if metadata.requires_python and not python_version_matches( + current, metadata.requires_python + ): + reasons.append( + f"Python {current} does not satisfy " + f"requires-python {metadata.requires_python!r}" + ) + if missing := missing_dependencies(metadata.dependencies): + reasons.append("missing dependencies: " + ", ".join(missing)) + return reasons + + +def _groundhog_with_spec() -> str: + """The ``--with`` spec that installs this same groundhog into a bootstrapped env.""" + if groundhog_hpc.__version__ == "0.0.0" and groundhog_hpc.__file__: + # No package metadata (e.g. a checkout without git tags): use the checkout. + return str(Path(groundhog_hpc.__file__).resolve().parents[2]) + return get_groundhog_version_spec() + + +def build_bootstrap_command( + script_path: Path, + harness: str, + harness_args: list[str], + metadata: Pep723Metadata, +) -> list[str]: + """Compose the ``uv run ... hog run ...`` command that re-runs a harness in an + environment built from the script's PEP 723 metadata. + + The ``[tool.uv]`` resolution settings a script can set (``exclude-newer``, + ``index-url``, ``extra-index-url``) are forwarded when present so the + driver resolves the same way the remote environment does. + + Args: + script_path: Resolved path to the user script + harness: Name of the harness to run + harness_args: Arguments for the harness (placed after ``--``) + metadata: The script's parsed PEP 723 metadata + + Returns: + The command as an argv list + """ + cmd = [ + uv.find_uv_bin(), + "run", + "--no-project", + "--python", + metadata.requires_python, + "--with", + _groundhog_with_spec(), + ] + for dep in metadata.dependencies: + cmd += ["--with", dep] + + uv_settings = metadata.tool.uv if metadata.tool is not None else None + if uv_settings is not None: + if uv_settings.exclude_newer: + cmd += ["--exclude-newer", uv_settings.exclude_newer] + if uv_settings.index_url: + cmd += ["--index-url", uv_settings.index_url] + for url in uv_settings.extra_index_url or []: + cmd += ["--extra-index-url", url] + + cmd += ["hog", "run", str(script_path), harness] + if harness_args: + cmd += ["--", *harness_args] + return cmd + + def check_and_update_metadata(script_path: Path, contents: str) -> str: """Check for missing/incomplete PEP 723 metadata and offer to update. diff --git a/tests/test_app.py b/tests/test_app.py index 60d99b6..a1a7782 100644 --- a/tests/test_app.py +++ b/tests/test_app.py @@ -3,6 +3,7 @@ import tempfile from pathlib import Path +import pytest from typer.testing import CliRunner from groundhog_hpc.app.main import app @@ -897,3 +898,232 @@ def main(dataset: str, epochs: int = 10): output = strip_ansi(result.output) assert "DATASET" in output or "dataset" in output.lower() assert "--epochs" in output + + +class TestMissingDependencies: + """missing_dependencies() decides whether hog run must bootstrap an environment.""" + + def test_satisfied_requirements_are_not_reported(self): + from groundhog_hpc.app.utils import missing_dependencies + + assert missing_dependencies(["typer>=0.1", "pydantic"]) == [] + + def test_missing_distribution_is_reported(self): + from groundhog_hpc.app.utils import missing_dependencies + + assert missing_dependencies(["definitely-not-installed-xyz"]) == [ + "definitely-not-installed-xyz" + ] + + def test_version_outside_specifier_is_reported(self): + from groundhog_hpc.app.utils import missing_dependencies + + assert missing_dependencies(["typer<0.0.1"]) == ["typer<0.0.1"] + + def test_false_marker_is_skipped(self): + from groundhog_hpc.app.utils import missing_dependencies + + dep = "definitely-not-installed-xyz; python_version < '3'" + assert missing_dependencies([dep]) == [] + + def test_url_requirement_checks_presence_only(self): + from groundhog_hpc.app.utils import missing_dependencies + + assert missing_dependencies(["typer @ git+https://example.invalid/typer"]) == [] + assert missing_dependencies( + ["definitely-not-installed-xyz @ git+https://example.invalid/x"] + ) == ["definitely-not-installed-xyz @ git+https://example.invalid/x"] + + def test_malformed_requirement_is_reported(self): + from groundhog_hpc.app.utils import missing_dependencies + + assert missing_dependencies(["not a requirement !!"]) == [ + "not a requirement !!" + ] + + +HARNESS_THAT_PRINTS = """ +import groundhog_hpc as hog + +@hog.harness() +def main(count: int = 1): + print(f"ran in-process:{count}") +""" + + +class TestRunBootstrap: + """hog run re-executes itself in a uv env when the current one can't host the harness.""" + + @pytest.fixture + def fake_subprocess_run(self, monkeypatch): + import subprocess + + calls = [] + + def _fake_run(cmd, env=None, **kwargs): + calls.append({"cmd": cmd, "env": env}) + return subprocess.CompletedProcess(cmd, 7) + + monkeypatch.setattr("groundhog_hpc.app.run.subprocess.run", _fake_run) + monkeypatch.delenv("GROUNDHOG_BOOTSTRAPPED", raising=False) + monkeypatch.delenv("GROUNDHOG_NO_BOOTSTRAP", raising=False) + return calls + + def test_bootstraps_when_dependencies_missing( + self, pep723_script, fake_subprocess_run + ): + script = pep723_script( + dependencies=["definitely-not-installed-xyz"], + extra_content=HARNESS_THAT_PRINTS, + ) + result = runner.invoke(app, ["run", str(script), "main", "--", "--count=5"]) + + assert result.exit_code == 7, result.output # child's return code propagates + assert "ran in-process" not in result.output + assert "Bootstrapping an environment" in result.output + assert "definitely-not-installed-xyz" in result.output + + (call,) = fake_subprocess_run + cmd = call["cmd"] + assert cmd[1:3] == ["run", "--no-project"] + assert cmd[cmd.index("--python") + 1] == ">=3.10" + assert "definitely-not-installed-xyz" in cmd + assert cmd[cmd.index("definitely-not-installed-xyz") - 1] == "--with" + assert cmd[-6:] == [ + "hog", + "run", + str(script.resolve()), + "main", + "--", + "--count=5", + ] + assert call["env"]["GROUNDHOG_BOOTSTRAPPED"] == "1" + + def test_bootstraps_on_python_mismatch(self, pep723_script, fake_subprocess_run): + script = pep723_script( + requires_python="==2.7.*", extra_content=HARNESS_THAT_PRINTS + ) + result = runner.invoke(app, ["run", str(script)]) + + assert result.exit_code == 7, result.output + assert "requires-python" in result.output + (call,) = fake_subprocess_run + cmd = call["cmd"] + assert cmd[cmd.index("--python") + 1] == "==2.7.*" + assert cmd[-4:] == ["hog", "run", str(script.resolve()), "main"] + assert "--" not in cmd # no harness args, no separator + + def test_runs_in_process_when_environment_satisfies( + self, pep723_script, fake_subprocess_run + ): + script = pep723_script( + dependencies=["typer"], extra_content=HARNESS_THAT_PRINTS + ) + result = runner.invoke(app, ["run", str(script), "--", "--count=3"]) + + assert result.exit_code == 0, result.output + assert "ran in-process:3" in result.output + assert fake_subprocess_run == [] + + def test_no_recursion_when_already_bootstrapped( + self, pep723_script, fake_subprocess_run, monkeypatch + ): + monkeypatch.setenv("GROUNDHOG_BOOTSTRAPPED", "1") + script = pep723_script( + dependencies=["definitely-not-installed-xyz"], + extra_content=HARNESS_THAT_PRINTS, + ) + result = runner.invoke(app, ["run", str(script)]) + + assert result.exit_code == 0, result.output + assert "ran in-process:1" in result.output + assert "Warning: running in the current environment" in result.output + assert fake_subprocess_run == [] + + def test_no_bootstrap_flag_runs_in_process( + self, pep723_script, fake_subprocess_run + ): + script = pep723_script( + dependencies=["definitely-not-installed-xyz"], + extra_content=HARNESS_THAT_PRINTS, + ) + result = runner.invoke(app, ["run", "--no-bootstrap", str(script)]) + + assert result.exit_code == 0, result.output + assert "ran in-process:1" in result.output + assert "Warning: running in the current environment" in result.output + assert fake_subprocess_run == [] + + def test_no_bootstrap_env_var_runs_in_process( + self, pep723_script, fake_subprocess_run, monkeypatch + ): + monkeypatch.setenv("GROUNDHOG_NO_BOOTSTRAP", "1") + script = pep723_script( + dependencies=["definitely-not-installed-xyz"], + extra_content=HARNESS_THAT_PRINTS, + ) + result = runner.invoke(app, ["run", str(script)]) + + assert result.exit_code == 0, result.output + assert "ran in-process:1" in result.output + assert fake_subprocess_run == [] + + +class TestBuildBootstrapCommand: + """Only user-set [tool.uv] settings are forwarded to the bootstrap command.""" + + def test_forwards_user_set_uv_settings(self, tmp_path): + from groundhog_hpc.app.utils import build_bootstrap_command + + contents = """# /// script +# requires-python = ">=3.12,<3.13" +# dependencies = ["numpy", "pandas>=2"] +# +# [tool.uv] +# exclude-newer = "2026-01-01T00:00:00Z" +# index-url = "https://mirror.example/simple" +# extra-index-url = ["https://download.pytorch.org/whl/cpu"] +# /// +""" + script = tmp_path / "s.py" + script.write_text(contents) + cmd = build_bootstrap_command( + script, "train", ["data", "--epochs=2"], read_pep723(contents) + ) + + assert cmd[1:5] == ["run", "--no-project", "--python", ">=3.12,<3.13"] + assert cmd[5] == "--with" + assert cmd[6].startswith("groundhog-hpc") or Path(cmd[6]).exists() + assert cmd[7:11] == ["--with", "numpy", "--with", "pandas>=2"] + assert cmd[11:13] == ["--exclude-newer", "2026-01-01T00:00:00Z"] + assert cmd[13:15] == ["--index-url", "https://mirror.example/simple"] + assert cmd[15:17] == [ + "--extra-index-url", + "https://download.pytorch.org/whl/cpu", + ] + assert cmd[17:] == [ + "hog", + "run", + str(script), + "train", + "--", + "data", + "--epochs=2", + ] + + def test_absent_uv_settings_are_not_forwarded(self, tmp_path): + from groundhog_hpc.app.utils import build_bootstrap_command + + contents = """# /// script +# requires-python = ">=3.12" +# dependencies = [] +# /// +""" + script = tmp_path / "s.py" + script.write_text(contents) + + cmd = build_bootstrap_command(script, "main", [], read_pep723(contents)) + assert "--exclude-newer" not in cmd + assert "--index-url" not in cmd + assert "--extra-index-url" not in cmd + assert cmd[-4:] == ["hog", "run", str(script), "main"]