hog run: bootstrap a driver environment from the script's PEP 723 header

docs/api/cli.md

@@ -27,6 +27,10 @@ Run a Python script on a Globus Compute endpoint.
 Use -- to pass arguments to parameterized harnesses:
     hog run script.py harness -- arg1 --option=value
 
+If the current environment does not satisfy the script's PEP 723
+requires-python / dependencies, hog run re-executes itself inside a uv
+environment built from that metadata (disable with --no-bootstrap).
+
 **Usage**:
 
 ```console
@@ -42,6 +46,7 @@ $ hog run [OPTIONS] SCRIPT [HARNESS]
 
 * `--no-fun-allowed`: Suppress emoji output
 * `--log-level TEXT`: Set logging level (DEBUG, INFO, WARNING, ERROR)
+* `--no-bootstrap`: Run the harness in the current environment even if it does not satisfy the script's requires-python / dependencies
 * `--help`: Show this message and exit.
 
 ## `hog init`

docs/api/environment_variables.md

@@ -84,6 +84,27 @@ export GROUNDHOG_CACHE_DIR=/gpfs/shared/uv-cache
 3. `$TMPDIR` (temporary directory)
 4. `/tmp` (system temp)
 
+## GROUNDHOG_NO_BOOTSTRAP
+
+**Type:** boolean (any truthy value)
+
+**Default:** not set
+
+Keeps `hog run` in the current environment even when that environment does not satisfy the script's PEP 723 `requires-python` / `dependencies`. Normally `hog run` re-executes itself inside a uv environment built from the script metadata in that case (see [Functions and Harnesses](../concepts/functions-and-harnesses.md#the-driver-environment)). Equivalent to the `--no-bootstrap` flag.
+
+**Example:**
+```bash
+GROUNDHOG_NO_BOOTSTRAP=1 hog run script.py
+```
+
+## GROUNDHOG_BOOTSTRAPPED
+
+**Type:** boolean (any truthy value)
+
+**Default:** not set
+
+Set automatically by `hog run` in the environment of the child process it bootstraps. A child that still finds the environment unsuitable warns and runs in-process instead of bootstrapping again. You should not need to set this yourself.
+
 ## `uv` Environment Variables
 
 Groundhog uses `uv` to manage Python environments on remote endpoints. Any `UV_*` environment variable can be used to override `[tool.uv]` configuration in your script.

docs/concepts/functions-and-harnesses.md

@@ -47,6 +47,20 @@ hog run script.py           # Runs the 'main' harness
 hog run script.py my_harness  # Runs a specific harness
 ```
 
+### The driver environment
+
+`hog run` imports your script in the process it runs in, so the harness (and anything defined at module level, such as classes that subclass a third-party base) needs its imports available *locally* as well as on the endpoint. The PEP 723 header already lists them, so `hog run` uses it: when the current interpreter does not satisfy `requires-python`, or any entry in `dependencies` is not installed, `hog run` re-executes itself inside a uv environment built from that metadata:
+
+```bash
+uv run --no-project --python "<requires-python>" \
+       --with groundhog-hpc==<this version> --with <dependency>... \
+       hog run script.py harness -- args
+```
+
+The `[tool.uv]` settings `exclude-newer`, `index-url` and `extra-index-url` are forwarded when the script sets them, so the driver resolves the same way the endpoint does. If the current environment already satisfies the header, nothing changes and the harness runs in-process.
+
+To keep the old in-process behavior, pass `--no-bootstrap` (before the script path) or set `GROUNDHOG_NO_BOOTSTRAP=1`; `hog run` then warns about the mismatch and continues.
+
 ### Parameterized harnesses
 
 Harnesses can accept parameters that map to CLI arguments. This makes harnesses reusable without editing code:

docs/examples/dependencies.md

@@ -41,7 +41,7 @@ def main():
 
 1. Dependencies are declared in the PEP 723 metadata block. These will be installed automatically by uv when the function runs on the remote endpoint. See also: [uv script dependencies](https://docs.astral.sh/uv/guides/scripts/#declaring-script-dependencies)
 
-2. Import remote dependencies **inside** the function, not at module level. This ensures imports happen after uv installs the packages in the remote environment.
+2. Import dependencies that only the remote side needs **inside** the function. The environment `hog run` starts from need not have them installed, and importing lazily keeps startup fast.
 
 ## Adding Dependencies
 
@@ -76,7 +76,9 @@ def process_data(data: list[float]) -> float:
     return np.std(data)
 ```
 
-**Why?** Module-level imports fail when the script loads locally (before uv installs packages on the remote endpoint). Function-level imports ensure the import happens after uv sets up the remote environment.
+**Why?** A module-level import runs wherever the script is loaded, including on your laptop when `hog run` imports it. If a package is missing there, `hog run` will bootstrap a driver environment from the PEP 723 header (see [the driver environment](../concepts/functions-and-harnesses.md#the-driver-environment)), which works but costs a resolve on every run. Importing inside the function keeps the driver light and lets the harness run in-process whenever it can.
+
+Module-level imports are the right call when the harness itself needs the package, for example a class defined in the script that subclasses a third-party base. Declare the package in `dependencies` and `hog run` takes care of the driver side.
 
 ## Running the Example
 

src/groundhog_hpc/app/run.py

@@ -2,22 +2,23 @@
 
 import inspect
 import os
+import subprocess
 import sys
 from pathlib import Path
 from typing import Any
 
 import typer
 
 from groundhog_hpc.app.utils import (
+    bootstrap_reasons,
+    build_bootstrap_command,
     check_and_update_metadata,
-    python_version_matches,
 )
 from groundhog_hpc.configuration.pep723 import read_pep723
 from groundhog_hpc.errors import RemoteExecutionError
 from groundhog_hpc.harness import Harness
 from groundhog_hpc.logging import setup_logging
 from groundhog_hpc.utils import (
-    get_groundhog_version_spec,
     import_user_script,
     path_to_module_name,
 )
@@ -71,11 +72,24 @@ def run(
         "--log-level",
         help="Set logging level (DEBUG, INFO, WARNING, ERROR)\n\n[env: GROUNDHOG_LOG_LEVEL=]",
     ),
+    no_bootstrap: bool = typer.Option(
+        False,
+        "--no-bootstrap",
+        help=(
+            "Run the harness in the current environment even if it does not "
+            "satisfy the script's requires-python / dependencies"
+            "\n\n[env: GROUNDHOG_NO_BOOTSTRAP=]"
+        ),
+    ),
 ) -> None:
     """Run a Python script on a Globus Compute endpoint.
 
     Use -- to pass arguments to parameterized harnesses:
         hog run script.py harness -- arg1 --option=value
+
+    If the current environment does not satisfy the script's PEP 723
+    requires-python / dependencies, hog run re-executes itself inside a uv
+    environment built from that metadata (disable with --no-bootstrap).
     """
     # Handle the -- separator for harness arguments
     # ctx.args may contain ['--', 'arg1', 'arg2'] - strip the '--' if present
@@ -108,28 +122,33 @@ def run(
     contents = check_and_update_metadata(script_path, contents)
 
     metadata = read_pep723(contents)
-    if metadata and metadata.requires_python:
-        requires_python = metadata.requires_python
-        current_version = f"{sys.version_info.major}.{sys.version_info.minor}.{sys.version_info.micro}"
-
-        if not python_version_matches(current_version, requires_python):
-            groundhog_spec = get_groundhog_version_spec()
-            uv_cmd = (
-                f"uv run --with {groundhog_spec} "
-                f"--python {requires_python} "
-                f"hog run {script_path} {harness}"
-            )
-
+    if metadata is not None and (reasons := bootstrap_reasons(metadata)):
+        skip_bootstrap = (
+            no_bootstrap
+            or bool(os.environ.get("GROUNDHOG_NO_BOOTSTRAP"))
+            or bool(os.environ.get("GROUNDHOG_BOOTSTRAPPED"))
+        )
+        if not skip_bootstrap:
+            # Re-run this same invocation inside a uv environment built from the
+            # script's metadata; the child sees GROUNDHOG_BOOTSTRAPPED and runs
+            # in-process, so a child that still fails the checks cannot loop.
+            cmd = build_bootstrap_command(script_path, harness, harness_args, metadata)
             typer.echo(
-                f"Warning: Script requires Python {requires_python}, "
-                f"but current version is {current_version}. This may "
-                "cause issues with serialization.",
+                f"Bootstrapping an environment from {script_path.name}'s "
+                f"metadata ({'; '.join(reasons)})",
                 err=True,
             )
-            typer.echo(
-                f"\nTo run with matching Python version, use:\n  {uv_cmd}",
-                err=True,
+            completed = subprocess.run(
+                cmd, env={**os.environ, "GROUNDHOG_BOOTSTRAPPED": "1"}
             )
+            raise typer.Exit(completed.returncode)
+
+        typer.echo(
+            "Warning: running in the current environment although "
+            f"{'; '.join(reasons)}. Imports may fail and remote results may "
+            "not deserialize.",
+            err=True,
+        )
 
     try:
         module_name = path_to_module_name(script_path)

src/groundhog_hpc/app/utils.py

@@ -1,20 +1,25 @@
 """Shared utility functions for the Groundhog CLI."""
 
+import importlib.metadata
 import subprocess
+import sys
 import tempfile
 from pathlib import Path
 
 import typer
 import uv
+from packaging.requirements import InvalidRequirement, Requirement
 from packaging.specifiers import InvalidSpecifier, SpecifierSet
 from packaging.version import Version
 
+import groundhog_hpc
 from groundhog_hpc.configuration.pep723 import (
     Pep723Metadata,
     insert_or_update_metadata,
     read_pep723,
     write_pep723,
 )
+from groundhog_hpc.utils import get_groundhog_version_spec
 
 
 def normalize_python_version_with_uv(python: str) -> str:
@@ -80,6 +85,127 @@ def python_version_matches(current: str, spec: str) -> bool:
     return Version(current) in SpecifierSet(spec)
 
 
+def current_python_version() -> str:
+    """Return the running interpreter's version as "X.Y.Z"."""
+    v = sys.version_info
+    return f"{v.major}.{v.minor}.{v.micro}"
+
+
+def missing_dependencies(dependencies: list[str]) -> list[str]:
+    """Return the entries of ``dependencies`` the current environment does not satisfy.
+
+    An entry is satisfied when a distribution with its name is installed and,
+    unless the entry points at a URL, the installed version is within its
+    specifier. Entries whose environment marker evaluates false are skipped.
+    Malformed entries are reported as missing so that uv gets to raise the
+    real error when the environment is built.
+
+    Args:
+        dependencies: PEP 508 requirement strings, e.g. from PEP 723 metadata
+
+    Returns:
+        The unsatisfied entries, verbatim, in their original order
+    """
+    missing: list[str] = []
+    for dep in dependencies:
+        try:
+            req = Requirement(dep)
+        except InvalidRequirement:
+            missing.append(dep)
+            continue
+        if req.marker is not None and not req.marker.evaluate():
+            continue
+        try:
+            installed = importlib.metadata.version(req.name)
+        except importlib.metadata.PackageNotFoundError:
+            missing.append(dep)
+            continue
+        if req.url is None and not req.specifier.contains(installed, prereleases=True):
+            missing.append(dep)
+    return missing
+
+
+def bootstrap_reasons(metadata: Pep723Metadata) -> list[str]:
+    """Explain why the current environment cannot host the script's harness.
+
+    Args:
+        metadata: The script's PEP 723 metadata
+
+    Returns:
+        Human-readable reasons; empty when the environment satisfies both
+        ``requires-python`` and ``dependencies``
+    """
+    reasons: list[str] = []
+    current = current_python_version()
+    if metadata.requires_python and not python_version_matches(
+        current, metadata.requires_python
+    ):
+        reasons.append(
+            f"Python {current} does not satisfy "
+            f"requires-python {metadata.requires_python!r}"
+        )
+    if missing := missing_dependencies(metadata.dependencies):
+        reasons.append("missing dependencies: " + ", ".join(missing))
+    return reasons
+
+
+def _groundhog_with_spec() -> str:
+    """The ``--with`` spec that installs this same groundhog into a bootstrapped env."""
+    if groundhog_hpc.__version__ == "0.0.0" and groundhog_hpc.__file__:
+        # No package metadata (e.g. a checkout without git tags): use the checkout.
+        return str(Path(groundhog_hpc.__file__).resolve().parents[2])
+    return get_groundhog_version_spec()
+
+
+def build_bootstrap_command(
+    script_path: Path,
+    harness: str,
+    harness_args: list[str],
+    metadata: Pep723Metadata,
+) -> list[str]:
+    """Compose the ``uv run ... hog run ...`` command that re-runs a harness in an
+    environment built from the script's PEP 723 metadata.
+
+    The ``[tool.uv]`` resolution settings a script can set (``exclude-newer``,
+    ``index-url``, ``extra-index-url``) are forwarded when present so the
+    driver resolves the same way the remote environment does.
+
+    Args:
+        script_path: Resolved path to the user script
+        harness: Name of the harness to run
+        harness_args: Arguments for the harness (placed after ``--``)
+        metadata: The script's parsed PEP 723 metadata
+
+    Returns:
+        The command as an argv list
+    """
+    cmd = [
+        uv.find_uv_bin(),
+        "run",
+        "--no-project",
+        "--python",
+        metadata.requires_python,
+        "--with",
+        _groundhog_with_spec(),
+    ]
+    for dep in metadata.dependencies:
+        cmd += ["--with", dep]
+
+    uv_settings = metadata.tool.uv if metadata.tool is not None else None
+    if uv_settings is not None:
+        if uv_settings.exclude_newer:
+            cmd += ["--exclude-newer", uv_settings.exclude_newer]
+        if uv_settings.index_url:
+            cmd += ["--index-url", uv_settings.index_url]
+        for url in uv_settings.extra_index_url or []:
+            cmd += ["--extra-index-url", url]
+
+    cmd += ["hog", "run", str(script_path), harness]
+    if harness_args:
+        cmd += ["--", *harness_args]
+    return cmd
+
+
 def check_and_update_metadata(script_path: Path, contents: str) -> str:
     """Check for missing/incomplete PEP 723 metadata and offer to update.