diff --git a/AGENTS.md b/AGENTS.md index 77248e0b..3410e93f 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -107,7 +107,7 @@ and deserialized through a registry. | Package | Contents | |---|---| -| `env/` | `Env` base class: a subclass declares `type`, implements `from_dict` and `async deploy(...) -> DeployedEnv`, and may override `reset`, `close` and `load_file_artifact_universe`; teardown closes each env a run deployed before terminating its sandboxes by id. Built-ins in `env/envs/`: `MCPServerEnv` (`mcp_server`), `MultiEnv` (`multi`, MCP servers plus websites), `ServiceDBEnv` (`service_db`, PostgreSQL), `WebsiteEnv` (`website`), `GatewayEnv` (`gateway_server`), `ComputerDriverEnv` (`computer_driver`, `env/envs/computer/driver_env.py`: a computer driver's files and what its class declares for each platform it supports, read at put in a process of its own, `agentenv_protocol.computer._declare`, where the driver's own dependencies needn't be installed, with the `port` it serves on and its install by OS, `[install.]`: files, a `python` requirements file for a venv, and the services the base's init runs; it deploys only as a computer env's driver), `ComputerEnv` (`computer`, in `env/envs/computer/`: a base computer and a driver, matched at put by gate 1 before anything is stored, then composed by `composer.py` into an image for each form and arch with this machine's Docker, the other arch through its emulation, tagged `agentenv-computers:` in the image store by a key of the image's Dockerfile and build context so any env with the same inputs reuses it; each pair's status goes into its metadata, `composed`, the arches built are written as its computer, a `computer` artifact over one index, and `agent-env env build` resumes what a put left; gate 3 then boots each native build on a configured sandbox provider and probes it at `full`, making it `ready` or `failed`, and a build no provider runs natively stays `unverified`. Its deploy, through the `computer` env provider (`providers/env_providers/env_computer_provider.py`), boots its first `ready` build on a sandbox provider that runs its arch natively (`native_platforms()`), with no gateway in front and the driver's card as the env card, then checks its `display`; `deploy_env` `env_params` take a `form`, `computer_settings` (deploy-phase values of the base's settings; a secret one, here or on the env, only as a bare `secret:`/`env:` reference) and `driver_settings` (the driver's `ComputerSettings`, such as its default coordinate space); a `network` that allows nothing runs it on local Docker's internal network behind a relay (`providers/sandbox_providers/local_docker/relay/`); `reset` and `close` have the driver end its recordings first (`computer-recording/v1` `end_all`, each uploading what's left and its manifest under the grant it started with), and `reset` then boots a fresh container in its place (`reset_container`), keeping the computer's address but not an agent's MCP session to the old driver; its `load_file_artifact_universe` writes files under the session user's home through the driver's `data/add`, `Downloads` by default, each file's bytes through a read grant when the store's grants reach it and its network allows everything, else inline (`env/envs/computer/utils/inside.py`)), `ComputerAppEnv` (`computer_app`, `env/envs/computer/app_env.py`: an app on a computer, a `ComputerAppEnvironment` class its env.toml's `module` names, read at put by the same `_declare`; deployed alone it is a one-app computer env, a `ComputerEnv` of the computer and driver its env.toml names, the composer installing it under `/opt/agentenv/apps//` with its services as supervisord programs beside its own server (`agentenv_protocol.computer.serve_app`, in a venv with the protocol's `app` extra, on `.agentenv/server.sock` in its folder), which adds its launcher, launches a `boot` app once per boot and says through `computer-app/v1` whether it's ready, as the driver's `computer-ready` asks, its card named after it, and `env_params` `app_settings` handing it values of its `Settings` in `AGENTENV_APP_SETTINGS`, a secret one only as a bare reference). An env type whose services take environment data defines `load_environment_artifact` and `load_environment_universe_artifact`; `load_artifact` refuses an environment artifact aimed at a type that doesn't, at save and before it loads. An env built on the protocol SDK can serve agents a skill (`@agent_skill`, `skill/v1`), which `add_skills` `env_ids` installs under its environment name (behind a gateway, each server's under that server's name); a skill with files beside its SKILL.md goes as a bundle stored once per content under `agentenv-env-skills/` in the object store. `env/gateway/gateway.py` is the env-side gateway server, container-only code. While its clock is armed it serves agents `get_time` (`env/gateway/get_time.py`), which answers in UTC and, given an IANA `time_zone`, also in that zone; executors' working `hours` (`env/gateway/triggers.py`) take a fixed `utc_offset` or an IANA `time_zone`. | +| `env/` | `Env` base class: a subclass declares `type`, implements `from_dict` and `async deploy(...) -> DeployedEnv`, and may override `reset`, `close` and `load_file_artifact_universe`; teardown closes each env a run deployed before terminating its sandboxes by id. Built-ins in `env/envs/`: `MCPServerEnv` (`mcp_server`), `MultiEnv` (`multi`, MCP servers plus websites), `ServiceDBEnv` (`service_db`, PostgreSQL), `WebsiteEnv` (`website`), `GatewayEnv` (`gateway_server`), `ComputerDriverEnv` (`computer_driver`, `env/envs/computer/driver_env.py`: a computer driver's files and what its class declares for each platform it supports, read at put in a process of its own, `agentenv_protocol.computer._declare`, where the driver's own dependencies needn't be installed, with the `port` it serves on and its install by OS, `[install.]` (`env/envs/computer/utils/install.py`): files, apt `packages` with extra `sources`, a `python` requirements file for a venv, `downloads` by URL and sha256, `run` scripts, the services the base's init runs, and `ports`; it deploys only as a computer env's driver), `ComputerEnv` (`computer`, in `env/envs/computer/`: a base computer and a driver, matched at put by gate 1 before anything is stored, then composed by `composer.py` into an image for each form and arch with this machine's Docker, the other arch through its emulation, tagged `agentenv-computers:` in the image store by a key of the image's Dockerfile, build context and lock so any env with the same inputs reuses it; every layer's packages, requirements and downloads are resolved at put in a container of the base for each arch (`utils/packages.py`, `utils/resolve_in_base.py`), stored in the object store by sha256 under `agentenv-packages/`, and listed in a lock stored by a digest of its inputs, which any env with the same inputs reuses, and builds install only the lock's files, every install step `RUN --network=none`; each pair's status goes into its metadata, `composed`, the arches built are written as its computer, a `computer` artifact over one index, and `agent-env env build` resumes what a put left; gate 3 then boots each native build on a configured sandbox provider and probes it at `full`, making it `ready` or `failed`, and a build no provider runs natively stays `unverified`. Its deploy, through the `computer` env provider (`providers/env_providers/env_computer_provider.py`), boots its first `ready` build on a sandbox provider that runs its arch natively (`native_platforms()`), with no gateway in front and the driver's card as the env card, then checks its `display`; `deploy_env` `env_params` take a `form`, `computer_settings` (deploy-phase values of the base's settings; a secret one, here or on the env, only as a bare `secret:`/`env:` reference) and `driver_settings` (the driver's `ComputerSettings`, such as its default coordinate space); a `network` that allows nothing runs it on local Docker's internal network behind a relay (`providers/sandbox_providers/local_docker/relay/`); `reset` and `close` have the driver end its recordings first (`computer-recording/v1` `end_all`, each uploading what's left and its manifest under the grant it started with), and `reset` then boots a fresh container in its place (`reset_container`), keeping the computer's address but not an agent's MCP session to the old driver; its `load_file_artifact_universe` writes files under the session user's home through the driver's `data/add`, `Downloads` by default, each file's bytes through a read grant when the store's grants reach it and its network allows everything, else inline (`env/envs/computer/utils/inside.py`)), `ComputerAppEnv` (`computer_app`, `env/envs/computer/app_env.py`: an app on a computer, a `ComputerAppEnvironment` class its env.toml's `module` names, read at put by the same `_declare`, which declares each OS with its own decorator (`@linux(...)`, or `@install(os_type, spec)` for any OS type) holding its install, its launch and, optionally, a skill and settings and build settings that add to the class's there, and install steps and ready checks for every OS or those their `os=` names; everything that reads its settings, launch or steps asks for the computer's OS type; deployed alone it is a one-app computer env, a `ComputerEnv` of the computer and driver its env.toml names, the composer installing it under `/opt/agentenv/apps//` with its services as supervisord programs beside its own server (`agentenv_protocol.computer.serve_app`, in a venv with the protocol's `app` extra, on `.agentenv/server.sock` in its folder), which adds its launcher, launches a `boot` app once per boot and says through `computer-app/v1` whether it's ready, as the driver's `computer-ready` asks, its card named after it, and `env_params` `app_settings` handing it values of its `Settings` in `AGENTENV_APP_SETTINGS`, a secret one only as a bare reference; its env.toml's `build_settings`, values of its `BuildSettings` checked at put, are written to `.agentenv/build_settings.json` in its folder, so they're part of the image's key, its install steps read them, and a deploy can't change them). An env type whose services take environment data defines `load_environment_artifact` and `load_environment_universe_artifact`; `load_artifact` refuses an environment artifact aimed at a type that doesn't, at save and before it loads. An env built on the protocol SDK can serve agents a skill (`@agent_skill`, `skill/v1`), which `add_skills` `env_ids` installs under its environment name (behind a gateway, each server's under that server's name); a skill with files beside its SKILL.md goes as a bundle stored once per content under `agentenv-env-skills/` in the object store. `env/gateway/gateway.py` is the env-side gateway server, container-only code. While its clock is armed it serves agents `get_time` (`env/gateway/get_time.py`), which answers in UTC and, given an IANA `time_zone`, also in that zone; executors' working `hours` (`env/gateway/triggers.py`) take a fixed `utc_offset` or an IANA `time_zone`. | | `artifact/` | Immutable versioned artifacts; every `put()` creates a new version. The document goes to the document store, binary payloads to the object store. Every artifact but an environment universe, whose `metadata` key holds its legacy metadata file ids, carries a `metadata` dict outside its versioned fields, changed by compare-and-swap (`merge_metadata`). Built-ins: `CliArtifact` (`cli`), `ComputerArtifact` (`computer`, a base computer: its architectures, OS, session user, settings and container form, whose image `put` reads from its registry into `metadata.validated_images`), `FileArtifact` (`file`), `FileArtifactUniverse` (`file_artifact_universe`), `DockerImageArtifact` (`docker_image`), `EnvironmentArtifact` (`environment`), `EnvironmentUniverseArtifact` (`environment_universe`), `SkillArtifact` (`skill`); `[artifacts] type_aliases` maps another stored spelling of a type onto one of them. A `DockerImageArtifact` holds a tar.gz of its image, or, written by `put_ref` (`--image-ref` on `env mcp-server put` and `a2a-agent put`), only a registry reference with its tag pinned to a digest, which sandboxes pull, or a published image's tag with its `image_digest` and its build context, which sandboxes pull by the digest and build from the context when they can't (the infra images, and a bundle agent built from a shipped agent's sources, `Plan.published`), or, written by `put_context`, only its build context, which a VM sandbox builds before running it (`VmSandbox.load_docker_images`) and Modal builds before creating its container (`ModalSandboxProvider.prepare_image`, outside a fallback chain's deadline); anything else that runs images by name (a container on the local provider, a lone server on any provider but Modal) refuses such an image. A build context (`utils/docker/context.py`) is what `docker build` would send, `.dockerignore` applied, written so equal files give equal bytes and an equal `source_digest`. | | `task/`, `task_step/` | A `Task` holds its `TaskStep`s inline; `Task.run()` executes them as a DAG. `depends_on` (None means all prior steps) gates a step, independent steps run concurrently, `fail_task_on_error` makes a failure fatal or tolerated, `retry_config` rolls a failed span back through the step journal and re-dispatches it. Built-in steps live in `task_step/task_steps/` (`deploy_env`, `deploy_agent`, `prompt_agent`, the verifiers under `verifiers/`, `start_computer_recording` and `end_computer_recording`, and more). On an env whose card serves computer-exec/v1 (a computer), `run_code` with `env_id` and `run_container_unit_tests_verifier` with `env_id` run inside it as its session user through the driver: files staged with `data/add` and read with `data/get`, commands run with `run`; `env_outcome_verifier` hands a `verify` that takes an `env_url` keyword the env's address. `teardown_run` and the `teardown_sandboxes` step share one core (`task/teardown.py` `tear_down`): the recordings the run left open end with fresh grants, each env still up closes (`Env.close`; a computer env ends any recording still open), what stayed open is registered from its namespace, then every sandbox is terminated by id. `task/episode/` runs a task with some of its agents played from outside, as a training loop does (`EpisodeSpec`). | | `store/` | Four store ABCs with local and cloud implementations: `DocumentStore` (SQLite, MongoDB, DynamoDB, Firestore with MongoDB compatibility), `ObjectStore` (filesystem, S3, Cloud Storage), `ImageStore` (local OCI registry, ECR), `SecretStore` (env vars or file, AWS Secrets Manager, Google Cloud Secret Manager). `VersionedEntityStore` implements the shared versioned get/put logic, `QueryBuilder` is the immutable chained query API, `store/base.py` holds the error types. `store/routing.py` is the `@local` namespace: while the CLI or a `build_app` server runs, an `@local/…` id and what an `@local` task's run records go to local stores beside the configured ones, and that run's writes to the configured stores are refused. A new backend must pass the conformance kits in `tst/store/`. | diff --git a/packages/agentenv-protocol/README.md b/packages/agentenv-protocol/README.md index 79290eeb..fe1176ed 100644 --- a/packages/agentenv-protocol/README.md +++ b/packages/agentenv-protocol/README.md @@ -615,37 +615,75 @@ server, which needs the `app` extra (below). ```python from pydantic import BaseModel, SecretStr -from agentenv_protocol.computer import ComputerAppEnvironment, Launch, Linux, Service, computer_app +from agentenv_protocol.computer import (ComputerAppEnvironment, Launch, Packages, Service, computer_app, + install_step, linux) + +class LinuxSettings(BaseModel): + font: str = "DejaVu Sans" @computer_app(supports=[{"os": {"type": "linux", "display": ["x11"]}, "arch": ["amd64", "arm64"]}]) +@linux(packages=Packages(apt=["thunderbird"]), python="requirements.txt", + services=[Service(name="imap", command="venv/bin/python imap.py")], + launch=Launch(command=["thunderbird"], window={"class": "thunderbird"}), settings=LinuxSettings) class MailApp(ComputerAppEnvironment): - linux = Linux(python="requirements.txt", services=[Service(name="imap", command="venv/bin/python imap.py")]) - launch = Launch(command=["thunderbird"], window={"class": "thunderbird"}) skill = "skill.md" class Settings(BaseModel): signature: str = "" password: SecretStr = SecretStr("") + + class BuildSettings(BaseModel): + locale: str = "en_US" + + @install_step(scope="session") + def profile(self) -> None: ... ``` - **`@computer_app`** names the platforms the app supports, as a driver names them, and the `permissions` it needs of the computer. Its env.toml names the rest: the app itself - (`environment_name`) and the `computer` and `driver` it deploys alone on. -- **`linux = Linux(...)`** is how a Linux computer installs it: `files`, a folder of the app's - copied under its prefix, which holds the class's file; `python`, a requirements file there, - installed into the venv every app gets beside them; `services`, programs the computer's init runs - from the prefix as the session user. -- **`launch = Launch(...)`** says how it starts in the desktop session: with it (`at="boot"`) or when + (`environment_name`), the `computer` and `driver` it deploys alone on, and `build_settings`, its + build settings' values (`build_settings = { locale = "fr_FR" }`). +- **`@linux(...)`** is how a Linux computer installs it, `Linux`'s fields: `files`, a folder of the app's + copied under its prefix, which holds the class's file; `packages`, the OS's, as apt takes them + (`Packages(apt=["name", "name=version", "name/suite"])`); `python`, a requirements file there, + installed into the venv every app gets beside them; `sources`, extra apt repositories + (`AptSource`, with a key file in the folder); `downloads`, files fetched by URL and checked by + sha256 (`Download`); `run`, scripts of the folder run as root or the session user (`RunScript`); + `services`, programs the computer's init runs from the prefix as the session user; and `ports`, + those it listens on. Packages, requirements and downloads are resolved and fetched once, when the + computer env is put, and every build installs only those stored files, with the network closed. + Each value is checked as the class is defined: a URL, a sha256, a path that stays in the folder. + Installs are exact: apt installs each package without its recommends, and the venv gets exactly + the wheels resolved for it (`pip install --no-deps`). A requirements file lists everything itself, + since its lock covers that file alone: a line including another file (`-r`, `-c`), an editable + install (`-e`), a local path or a local folder of wheels is refused at put. +- **Each OS has its own decorator,** and what differs by OS goes in it: `launch=` always; and + optionally `skill=`, a file used instead of the class's, and `settings=` and `build_settings=`, + models whose fields add to the class's `Settings` and `BuildSettings` on that OS. What's on the + class applies to every OS. `@install(os_type, spec, launch=..., ...)` is the generic form, for an + OS type a plugin adds, with that OS type's own install model as `spec`; `@linux(...)` is + `@install("linux", Linux(...), ...)`. On the computer the class reads what's declared for the OS + it runs on (`platform.detect()`). +- **`launch=Launch(...)`** says how it starts in the desktop session: with it (`at="boot"`) or when the agent opens it (`"demand"`). `ready=Ready(...)` says when it's ready, within `timeout_s` of its launch: a window matching `launch.window` by its class, instance or title, a port answering on loopback, or a command exiting 0. `@ready_check` marks a method for readiness that needs logic (a sync one runs in a thread), and `@install_step(scope=...)` one a computer runs as it installs the - app. + app, after its run scripts, offline and without the class's `__init__`. Each runs on every OS the + app installs on, or on those its `os=` names (`@install_step(os="linux")`, `@ready_check(os=[...])`); + naming one the app declares no install for is refused when the class is read. A step reads + `self.build_settings`; a deploy's `self.settings` aren't known at build, and reading one raises + `AttributeError` (`python -m agentenv_protocol.computer.run_install_step : `). - **Its skill** is a file of its folder (`skill = "skill.md"`) or an `@agent_skill` method, not both; its data hooks are the usual `@add_data`, `@get_data` and `@reset_data`. - **`Settings`** is typed; a computer hands each app the values a deploy gives it, as one JSON value in `APP_SETTINGS_ENV` (`AGENTENV_APP_SETTINGS`, the apps' values by name), and the base class reads its own, by `ENVIRONMENT_NAME`, as `self.settings`. A `SecretStr` field is a secret. +- **`BuildSettings`** is typed too: values the app's env fixes in its computer's image, which only + another image changes, so a deploy can't give one. The composer writes them to + `.agentenv/build_settings.json` in the app's folder, and the base class reads them, over the + defaults, as `self.build_settings`, in its install steps and on the computer alike. None is a + secret, since the image keeps them; `declaration()` refuses a `SecretStr` one. On a computer each app is its own server: `python -m agentenv_protocol.computer.serve_app :`, a program the computer's init runs from the app's folder in the app's venv, which diff --git a/packages/agentenv-protocol/src/agentenv_protocol/computer/__init__.py b/packages/agentenv-protocol/src/agentenv_protocol/computer/__init__.py index 6408b55e..5ffb4ed0 100644 --- a/packages/agentenv-protocol/src/agentenv_protocol/computer/__init__.py +++ b/packages/agentenv-protocol/src/agentenv_protocol/computer/__init__.py @@ -11,13 +11,20 @@ from .app import ( APP_SETTINGS_ENV, AppDeclaration, + AppOs, + AptSource, ComputerAppEnvironment, + Download, Launch, Linux, + Packages, Ready, + RunScript, Service, computer_app, + install, install_step, + linux, ready_check, ) from .errors import ( @@ -97,8 +104,9 @@ def __getattr__(name: str) -> Any: __all__ = [ "ComputerDriverEnvironment", "computer_driver", "DEFAULT_IDENTITY", "DriverDeclaration", - "ComputerAppEnvironment", "computer_app", "install_step", "ready_check", "Linux", "Service", "Launch", "Ready", - "AppDeclaration", + "ComputerAppEnvironment", "computer_app", "install", "linux", "install_step", "ready_check", "Linux", "Service", + "Launch", "Ready", "Packages", "AptSource", "Download", "RunScript", + "AppDeclaration", "AppOs", "APP_SETTINGS_ENV", "Display", "Image", "Effect", "ExecResult", "ReadyCheck", "Platform", "ComputerSettings", "SETTINGS_ENV", "PRINTABLE_KEYS", diff --git a/packages/agentenv-protocol/src/agentenv_protocol/computer/_declare.py b/packages/agentenv-protocol/src/agentenv_protocol/computer/_declare.py index c0c043d4..cf245899 100644 --- a/packages/agentenv-protocol/src/agentenv_protocol/computer/_declare.py +++ b/packages/agentenv-protocol/src/agentenv_protocol/computer/_declare.py @@ -98,7 +98,8 @@ def main(argv: list[str]) -> int: with contextlib.redirect_stdout(sys.stderr): declared = declare(Path(folder), file, name) except ValidationError as e: - print("ValueError: " + "; ".join(error["msg"].removeprefix("Value error, ") for error in e.errors()), + print("ValueError: " + "; ".join(f"{'.'.join([e.title, *map(str, error['loc'])])}: " + f"{error['msg'].removeprefix('Value error, ')}" for error in e.errors()), file=sys.stderr) return 1 except Exception as e: diff --git a/packages/agentenv-protocol/src/agentenv_protocol/computer/linux.py b/packages/agentenv-protocol/src/agentenv_protocol/computer/_linux.py similarity index 100% rename from packages/agentenv-protocol/src/agentenv_protocol/computer/linux.py rename to packages/agentenv-protocol/src/agentenv_protocol/computer/_linux.py diff --git a/packages/agentenv-protocol/src/agentenv_protocol/computer/app.py b/packages/agentenv-protocol/src/agentenv_protocol/computer/app.py index 488ffcc8..25308532 100644 --- a/packages/agentenv-protocol/src/agentenv_protocol/computer/app.py +++ b/packages/agentenv-protocol/src/agentenv_protocol/computer/app.py @@ -1,23 +1,27 @@ """ComputerAppEnvironment: software on a computer, an AgentEnvEnvironment whose class also declares what it is for a -computer: the platforms it supports and the permissions it needs (``@computer_app``), how each OS installs it -(``linux = Linux(...)``), how it launches (``launch``), its install steps and ready checks, its skill, and its typed -``Settings``. A ``computer_app`` env's put reads that declaration from the class; its data hooks are the base class's. -Its name is its env's ``environment_name``. On the computer the class runs as the app's own server (``serve_app``), -which launches the app, adds its launcher and says whether it's ready (``computer-app/v1``).""" +computer: the platforms it supports and the permissions it needs (``@computer_app``); for each OS, how it installs and +launches there and what differs there (``@linux``, or ``@install`` for any OS type); its install steps and ready checks, +each on every OS or on those it names; its skill, its typed ``Settings`` and its ``BuildSettings``. A ``computer_app`` +env's put reads that declaration from the class; its data hooks are the base class's. Its name is its env's +``environment_name``. On the computer the class runs as the app's own server (``serve_app``), which launches the app, +adds its launcher and says whether it's ready (``computer-app/v1``), as declared for the OS it runs on.""" from __future__ import annotations +import functools import inspect import json import os -from pathlib import Path -from typing import Any, Callable, ClassVar, Literal, Optional +from dataclasses import dataclass +from pathlib import Path, PurePosixPath +from typing import Annotated, Any, Callable, ClassVar, Literal, Optional, Union -from pydantic import BaseModel, ConfigDict, Field, field_validator, model_validator +from pydantic import AfterValidator, BaseModel, ConfigDict, Field, create_model, field_validator, model_validator from ..agent_env_environment import _EXT_ATTR, _OP_ATTR, AgentEnvEnvironment, extension from ..types import METHOD_ADD, METHOD_GET, METHOD_RESET, SKILL_EXTENSION_URI from .app_runtime import AppRuntime from .errors import InvalidArgument +from .platform import detect #: The environment variable a computer delivers its apps' settings in: JSON of each app's values by its name. APP_SETTINGS_ENV = "AGENTENV_APP_SETTINGS" @@ -31,10 +35,26 @@ #: Where an app's own server listens, in the app's folder, and where it says why it last failed to start. SOCKET = Path(".agentenv/server.sock") FAILURE = Path(".agentenv/server.failure") +#: The values of an app's build settings its env fixes in its computer's image, in the app's folder. +BUILD_SETTINGS = Path(".agentenv/build_settings.json") +# The JSON schema keywords that hold a subschema, a list of them, or a map of them by name. +_SUBSCHEMA = ("items", "additionalProperties", "not") +_SUBSCHEMA_LISTS = ("anyOf", "allOf", "oneOf", "prefixItems") +_SUBSCHEMA_MAPS = ("properties", "$defs", "patternProperties") #: How long an app may take to be ready when its launch doesn't say. DEFAULT_TIMEOUT_S = 60.0 +def _inside(path: str) -> str: + if not path or PurePosixPath(path).is_absolute() or ".." in PurePosixPath(path).parts: + raise ValueError(f"{path!r} must be a path inside the layer's folder") + return path + + +#: A path relative to the folder an app or a driver installs from, which never leaves it. +LayerPath = Annotated[str, AfterValidator(_inside)] + + class Service(BaseModel): """A program the computer's init runs from the app's folder as the session user; a relative program names one there.""" @@ -45,15 +65,72 @@ class Service(BaseModel): command: str +class Packages(BaseModel): + """The OS's packages, each as its package manager takes it: for apt, ``name``, ``name=version`` or + ``name/suite``.""" + + model_config = ConfigDict(extra="forbid") + + apt: list[str] = Field(default_factory=list) + + @field_validator("apt") + @classmethod + def _one_word_each(cls, value: list[str]) -> list[str]: + if bad := [name for name in value if not name or name.split() != [name] or name.startswith("-")]: + raise ValueError(f"each package is one word, as apt takes it, not {bad}") + return value + + +class AptSource(BaseModel): + """An apt repository resolved from besides the OS's own: its URL, suite and components, and its signing key, a file + of the folder it installs from.""" + + model_config = ConfigDict(extra="forbid") + + apt: str = Field(pattern=r"^https?://") + suite: str = Field(min_length=1) + components: list[str] = Field(default_factory=list) + key: LayerPath + + +class Download(BaseModel): + """A file fetched by its URL and checked against its sha256, put at ``to`` under the prefix it installs to; for one + arch only, when ``arch`` names it.""" + + model_config = ConfigDict(extra="forbid") + + url: str = Field(pattern=r"^https?://") + sha256: str = Field(pattern=r"^[0-9a-f]{64}$") + to: LayerPath + arch: Optional[Literal["amd64", "arm64"]] = None + + +class RunScript(BaseModel): + """A script of the folder it installs from, run as root, or for ``session``, as the session user, once its files, + packages and venv are in.""" + + model_config = ConfigDict(extra="forbid") + + script: LayerPath + scope: Literal["system", "session"] = "system" + + class Linux(BaseModel): - """How a Linux computer installs the app: ``files``, a folder of the app's copied under its prefix; ``python``, a - requirements file in that folder installed into a venv beside them; ``services``, programs its init runs.""" + """How a Linux computer installs the app: ``files``, a folder of the app's copied under its prefix; ``packages``, + the OS's; ``python``, a requirements file in that folder installed into a venv beside them; ``sources``, extra apt + repositories; ``downloads``, files fetched by URL; ``run``, scripts run once those are in; ``services``, programs + its init runs; ``ports``, those it listens on. Everything is fetched once at put, and builds install only that.""" model_config = ConfigDict(extra="forbid") - files: str = "." - python: Optional[str] = None + files: LayerPath = "." + packages: Optional[Packages] = None + python: Optional[LayerPath] = None + sources: Optional[list[AptSource]] = None + downloads: Optional[list[Download]] = None + run: Optional[list[RunScript]] = None services: list[Service] = Field(default_factory=list) + ports: Optional[list[int]] = None class Ready(BaseModel): @@ -109,30 +186,81 @@ class AppSkill(BaseModel): class InstallStep(BaseModel): - """An ``@install_step`` method, by name, and whom it runs as.""" + """An ``@install_step`` method, by name, whom it runs as, and the OS types it runs on: every one when ``os`` is + None.""" model_config = ConfigDict(extra="forbid") name: str scope: Literal["system", "session"] + os: Optional[list[str]] = None + + +class AppReadyCheck(BaseModel): + """A ``@ready_check`` method, by name, and the OS types it runs on: every one when ``os`` is None.""" + + model_config = ConfigDict(extra="forbid") + + name: str + os: Optional[list[str]] = None + + +class AppOs(BaseModel): + """What an app declares for one OS type: ``install``, how that OS installs it, as the OS type's install model gives + it; its ``launch`` there; and its ``skill``, ``settings`` and ``build_settings`` there when they differ from the + class's, the schemas with the class's fields added.""" + + model_config = ConfigDict(extra="forbid") + + install: dict[str, Any] + launch: Optional[Launch] = None + skill: Optional[AppSkill] = None + settings: Optional[dict[str, Any]] = None + build_settings: Optional[dict[str, Any]] = None class AppDeclaration(BaseModel): """What a ComputerAppEnvironment class declares, as a ``computer_app`` env stores it: the platforms it ``supports``, - as a driver names them; its ``install`` by OS type; ``settings``, the JSON schema of its ``Settings``; and the - ``data`` methods, ``install_steps`` and ``ready_checks`` it serves, by name.""" + as a driver names them; ``os``, what it declares for each OS type; ``settings`` and ``build_settings``, the JSON + schemas of its ``Settings`` and of its ``BuildSettings`` when it declares any; its ``skill``; and the ``data`` + methods, ``install_steps`` and ``ready_checks`` it serves, by name.""" model_config = ConfigDict(extra="forbid") supports: list[dict[str, Any]] permissions: list[str] - install: dict[str, Linux] - launch: Optional[Launch] = None + os: dict[str, AppOs] settings: dict[str, Any] + build_settings: Optional[dict[str, Any]] = None skill: Optional[AppSkill] = None data: list[str] install_steps: list[InstallStep] - ready_checks: list[str] + ready_checks: list[AppReadyCheck] + + @model_validator(mode="before") + @classmethod + def _stored_before_each_os(cls, data: Any) -> Any: + """A declaration stored before each OS had its own part: its ``install`` by OS type and one ``launch`` for + them, which become ``os``, and its ready checks by name.""" + if not isinstance(data, dict) or "install" not in data: + return data + data = dict(data) + installs, launch = data.pop("install"), data.pop("launch", None) + data["os"] = {os_type: {"install": install, "launch": launch} for os_type, install in installs.items()} + data["ready_checks"] = [{"name": check} if isinstance(check, str) else check + for check in data.get("ready_checks", [])] + return data + + +@dataclass(frozen=True) +class _OsDeclared: + """What ``@install`` declares for one OS type, as the class keeps it.""" + + install: BaseModel + launch: Optional[Launch] + skill: Optional[str] + settings: Optional[type[BaseModel]] + build_settings: Optional[type[BaseModel]] def computer_app(*, supports: Optional[list] = None, permissions: Optional[list[str]] = None) -> Callable: @@ -146,45 +274,101 @@ def deco(cls: type) -> type: return deco -def install_step(scope: Literal["system", "session"] = "system") -> Callable: - """Mark a method a computer runs as it installs the app, after its files, in the order the class defines them: as - root, or for ``session``, as the session user.""" +def install(os_type: str, spec: BaseModel, *, launch: Optional[Launch] = None, skill: Optional[str] = None, + settings: Optional[type[BaseModel]] = None, + build_settings: Optional[type[BaseModel]] = None) -> Callable: + """Declare how a computer of ``os_type`` installs the app, ``spec`` being that OS type's install model, and what is + the app's own there: its ``launch``; its ``skill``, a file of its folder, instead of the class's; and ``settings`` + and ``build_settings``, models whose fields add to the class's ``Settings`` and ``BuildSettings`` there. ``@linux`` + is this for Linux; an OS type a plugin adds is declared with it.""" + def deco(cls: type) -> type: + if not (isinstance(cls, type) and issubclass(cls, ComputerAppEnvironment)): + raise TypeError("@install decorates a ComputerAppEnvironment subclass") + cls._installs = {**cls._installs, os_type: _OsDeclared(spec, launch, skill, settings, build_settings)} + return cls + return deco + + +def linux(*, files: str = ".", packages: Optional[Packages] = None, python: Optional[str] = None, + sources: Optional[list[AptSource]] = None, downloads: Optional[list[Download]] = None, + run: Optional[list[RunScript]] = None, services: Optional[list[Service]] = None, + ports: Optional[list[int]] = None, launch: Optional[Launch] = None, skill: Optional[str] = None, + settings: Optional[type[BaseModel]] = None, build_settings: Optional[type[BaseModel]] = None) -> Callable: + """``@install`` for Linux: how a Linux computer installs the app, ``Linux``'s fields, and what is its own there.""" + return install("linux", Linux(files=files, packages=packages, python=python, sources=sources, downloads=downloads, + run=run, services=services or [], ports=ports), + launch=launch, skill=skill, settings=settings, build_settings=build_settings) + + +def install_step(scope: Literal["system", "session"] = "system", *, + os: Union[str, list[str], None] = None) -> Callable: + """Mark a method a computer runs as it installs the app, after its files, packages, venv and run scripts, in the + order the class defines them: as root, or for ``session``, as the session user; on every OS the app installs on, + or on the ``os`` types named. It runs offline, at build, without the class's ``__init__``, on + ``self.build_settings``; a deploy's ``self.settings`` aren't known yet.""" def deco(fn: Callable) -> Callable: - setattr(fn, _INSTALL_STEP_ATTR, scope) + setattr(fn, _INSTALL_STEP_ATTR, (scope, _named(os))) return fn return deco -def ready_check(fn: Callable) -> Callable: - """Mark a method that says whether the app is ready, when ``launch``'s ``ready`` can't.""" - setattr(fn, _READY_CHECK_ATTR, True) - return fn +def ready_check(fn: Optional[Callable] = None, *, os: Union[str, list[str], None] = None) -> Callable: + """Mark a method that says whether the app is ready, when its launch's ``ready`` can't: on every OS the app + installs on, or on the ``os`` types named; ``@ready_check`` or ``@ready_check(os=...)``.""" + def mark(fn: Callable) -> Callable: + setattr(fn, _READY_CHECK_ATTR, _named(os)) + return fn + return mark(fn) if fn is not None else mark + + +def _named(os_types: Union[str, list[str], None]) -> Optional[list[str]]: + return None if os_types is None else [os_types] if isinstance(os_types, str) else list(os_types) class ComputerAppEnvironment(AgentEnvEnvironment): - """Subclass it, declare it with ``@computer_app``, and set ``linux``, ``launch`` and ``skill`` as class attributes; - a deploy's settings arrive as ``self.settings``.""" + """Subclass it, declare it with ``@computer_app`` and an OS decorator for each OS (``@linux``), and set ``skill`` as + a class attribute; a deploy's settings arrive as ``self.settings``, those its env fixes at build as + ``self.build_settings``, and its launch as ``self.launch``, each as declared for the OS it runs on.""" supports: ClassVar[list] = [] permissions: ClassVar[list[str]] = [] + _installs: ClassVar[dict[str, _OsDeclared]] = {} _computer_app: Optional[AppRuntime] = None - linux: ClassVar[Optional[Linux]] = None - launch: ClassVar[Optional[Launch]] = None + launch: Optional[Launch] = None #: A file of the app's folder holding its skill for agents, when no ``@agent_skill`` method writes it. skill: ClassVar[Optional[str]] = None class Settings(BaseModel): """The app's settings: an app declares its own; a ``SecretStr`` field is a secret.""" + class BuildSettings(BaseModel): + """The app's build settings, which its env fixes in its computer's image and only another image changes: an + app declares its own, which its install steps read too. None is a secret, since the image keeps them.""" + def __init__(self) -> None: + cls, on = type(self), detect().os values = json.loads(os.environ.get(APP_SETTINGS_ENV) or "{}").get(os.environ.get("ENVIRONMENT_NAME"), {}) - self.settings = self.Settings.model_validate(values) + self.settings = cls.settings_on(on).model_validate(values) + self.build_settings = _built_settings(cls, on) + self.launch = declared.launch if (declared := cls._installs.get(on)) is not None else None ready = self.launch.ready if self.launch is not None else None self._computer_app = AppRuntime( - self.launch, {fn.__name__: getattr(self, fn.__name__) for fn in _methods(type(self)) - if hasattr(fn, _READY_CHECK_ATTR)}, + self.launch, {fn.__name__: getattr(self, fn.__name__) for fn in _methods(cls) + if hasattr(fn, _READY_CHECK_ATTR) and _runs_on(getattr(fn, _READY_CHECK_ATTR), on)}, ready.timeout_s if ready is not None else DEFAULT_TIMEOUT_S) + @classmethod + def settings_on(cls, os_type: Optional[str]) -> type[BaseModel]: + """Its ``Settings`` on ``os_type``, with the fields its OS adds.""" + added = declared.settings if (declared := cls._installs.get(os_type)) is not None else None + return _added(added, cls.Settings) if added is not None else cls.Settings + + @classmethod + def build_settings_on(cls, os_type: Optional[str]) -> type[BaseModel]: + """Its ``BuildSettings`` on ``os_type``, with the fields its OS adds.""" + added = declared.build_settings if (declared := cls._installs.get(os_type)) is not None else None + return _added(added, cls.BuildSettings) if added is not None else cls.BuildSettings + @extension(COMPUTER_APP, op="launch", description=( "The app's own server: launch launches the app from its folder, as its launcher and a boot do; ready says " "whether it's ready, by its launch's ready and its @ready_check methods.")) @@ -209,29 +393,87 @@ def extension_params(self, uri: str) -> dict: @classmethod def declaration(cls) -> AppDeclaration: - """What the class declares. Raises ValueError for a skill declared twice or an install that isn't a - ``Linux``.""" - if cls.linux is not None and not isinstance(cls.linux, Linux): - raise ValueError(f"{cls.__name__}.linux is a Linux(...), not {cls.linux!r}") + """What the class declares. Raises ValueError for a skill declared twice, a build setting that's a secret, or + an install step or ready check on an OS type the class declares no install for.""" + build = cls.BuildSettings.model_json_schema() + for name, schema in [(f"{cls.__name__}.BuildSettings", build), + *((declared.build_settings.__name__, declared.build_settings.model_json_schema()) + for declared in cls._installs.values() if declared.build_settings is not None)]: + if (secret := _secret_at(schema)) is not None: + raise ValueError(f"{name} holds a secret at {secret}, which its computer's image would keep; declare " + "it in Settings") methods = _methods(cls) skill_methods = [fn.__name__ for fn in methods if getattr(getattr(fn, _EXT_ATTR, None), "uri", None) == SKILL_EXTENSION_URI] if cls.skill is not None and skill_methods: raise ValueError(f"{cls.__name__} declares its skill twice: skill = {cls.skill!r} and @agent_skill " f"{skill_methods[0]}") + if skill_methods and (os_type := next((os_type for os_type, declared in cls._installs.items() + if declared.skill is not None), None)): + raise ValueError(f"{cls.__name__} declares its skill twice on {os_type}: skill = " + f"{cls._installs[os_type].skill!r} there and @agent_skill {skill_methods[0]}, which its " + "server serves on every OS") + steps = [InstallStep(name=fn.__name__, scope=scope, os=os_types) for fn in methods + if hasattr(fn, _INSTALL_STEP_ATTR) for scope, os_types in [getattr(fn, _INSTALL_STEP_ATTR)]] + checks = [AppReadyCheck(name=fn.__name__, os=getattr(fn, _READY_CHECK_ATTR)) for fn in methods + if hasattr(fn, _READY_CHECK_ATTR)] + for method in [*steps, *checks]: + if unknown := [os_type for os_type in method.os or [] if os_type not in cls._installs]: + raise ValueError(f"{cls.__name__}.{method.name} runs on {unknown[0]!r}, which {cls.__name__} declares " + f"no install for; it declares {', '.join(cls._installs) or 'none'}") return AppDeclaration( supports=cls.supports, permissions=cls.permissions, - install={"linux": cls.linux} if cls.linux is not None else {}, - launch=cls.launch, settings=cls.Settings.model_json_schema(), + os={os_type: AppOs( + install=declared.install.model_dump(mode="json", exclude_none=True), launch=declared.launch, + skill=AppSkill(file=declared.skill) if declared.skill is not None else None, + settings=cls.settings_on(os_type).model_json_schema() if declared.settings is not None else None, + build_settings=(cls.build_settings_on(os_type).model_json_schema() + if declared.build_settings is not None else None)) + for os_type, declared in cls._installs.items()}, + settings=cls.Settings.model_json_schema(), build_settings=build if build.get("properties") else None, skill=AppSkill(file=cls.skill) if cls.skill is not None else AppSkill(method=skill_methods[0]) if skill_methods else None, data=sorted({_DATA_METHODS[op] for fn in methods if (op := getattr(fn, _OP_ATTR, None)) in _DATA_METHODS}), - install_steps=[InstallStep(name=fn.__name__, scope=getattr(fn, _INSTALL_STEP_ATTR)) for fn in methods - if hasattr(fn, _INSTALL_STEP_ATTR)], - ready_checks=[fn.__name__ for fn in methods if hasattr(fn, _READY_CHECK_ATTR)], + install_steps=steps, ready_checks=checks, ) +def _secret_at(schema: dict, path: str = "") -> Optional[str]: + """Where in ``schema``, a JSON schema, a secret is, a ``SecretStr``'s ``writeOnly``: in it or any of its subschemas, + a field's, a list's items or a nested model's definition; None when it holds none. A default or a field's name + is data, never a marker.""" + if schema.get("writeOnly") is True: + return path + subschemas = [*((_joined(path, key), schema[key]) for key in _SUBSCHEMA if key in schema), + *((_joined(path, f"{key}[{index}]"), sub) for key in _SUBSCHEMA_LISTS + for index, sub in enumerate(schema.get(key, []))), + *((_joined(path, f"{key}.{name}"), sub) for key in _SUBSCHEMA_MAPS + for name, sub in schema.get(key, {}).items())] + return next((at for at in (_secret_at(sub, at) for at, sub in subschemas if isinstance(sub, dict)) + if at is not None), None) + + +def _joined(path: str, part: str) -> str: + return f"{path}.{part}" if path else part + + +def _built_settings(cls: type[ComputerAppEnvironment], os_type: Optional[str]) -> BaseModel: + """The app's build settings on ``os_type``, as its env fixed them in its computer's image, read in the app's folder: + the values the env gives, over the defaults.""" + values = json.loads(BUILD_SETTINGS.read_text()) if BUILD_SETTINGS.is_file() else {} + return cls.build_settings_on(os_type).model_validate(values) + + +def _runs_on(os_types: Optional[list[str]], os_type: Optional[str]) -> bool: + return os_types is None or os_type in os_types + + +@functools.cache +def _added(added: type[BaseModel], base: type[BaseModel]) -> type[BaseModel]: + """``base`` with the fields of ``added``, whose win where both declare one.""" + return create_model(added.__name__, __base__=(added, base)) + + def _methods(cls: type) -> list[Callable]: """The class's methods, each as the class resolves its name, in the order they were first defined.""" names = dict.fromkeys(name for klass in reversed(cls.__mro__) for name, member in vars(klass).items() diff --git a/packages/agentenv-protocol/src/agentenv_protocol/computer/driver.py b/packages/agentenv-protocol/src/agentenv_protocol/computer/driver.py index 068c2251..17067b56 100644 --- a/packages/agentenv-protocol/src/agentenv_protocol/computer/driver.py +++ b/packages/agentenv-protocol/src/agentenv_protocol/computer/driver.py @@ -10,7 +10,7 @@ from ..agent_env_environment import _CARD_CONFIG_ATTR, AgentEnvEnvironment, AgentEnvFastMCPApplication, environment_card from ..tool_context import ToolContext from ..types import EnvironmentCard, EnvironmentExtension, EnvironmentTool -from . import card, linux +from . import _linux, card from .checks import check_features, check_names from .dataplane import DataPlane from .extensions import Extensions @@ -149,11 +149,11 @@ def _computer_require_init(self) -> None: @default_for("run_in_session", os="linux") async def _computer_linux_run_in_session(self, argv: list, cwd: str | None, env: dict | None, timeout_s: float): - return await linux.run_in_session(argv, cwd, env, timeout_s, output_bytes=self.settings.exec_output_bytes) + return await _linux.run_in_session(argv, cwd, env, timeout_s, output_bytes=self.settings.exec_output_bytes) @default_for("write_files", os="linux") async def _computer_linux_write_files(self, files: dict) -> None: - await asyncio.to_thread(linux.write_files, files) + await asyncio.to_thread(_linux.write_files, files) def computer_driver(name: str | None = None, *, supports: list | None = None) -> Callable: diff --git a/packages/agentenv-protocol/src/agentenv_protocol/computer/run_install_step.py b/packages/agentenv-protocol/src/agentenv_protocol/computer/run_install_step.py new file mode 100644 index 00000000..bf78fae3 --- /dev/null +++ b/packages/agentenv-protocol/src/agentenv_protocol/computer/run_install_step.py @@ -0,0 +1,43 @@ +"""Run one of a computer app's ``@install_step`` methods, as its computer's build does: +``python -m agentenv_protocol.computer.run_install_step : ``, run from the app's venv in its +folder, offline. The class's ``__init__`` isn't run: the step has ``self.build_settings``, the values its env fixed, +and no deploy's ``self.settings``, which aren't known at build. Needs only the plain package.""" +from __future__ import annotations + +import asyncio +import inspect +import sys +from pathlib import Path +from typing import NoReturn + +from ._load import load_class +from .app import ComputerAppEnvironment, _built_settings +from .platform import detect + +USAGE = "usage: python -m agentenv_protocol.computer.run_install_step : " + + +class _NoDeploy: + """What ``self.settings`` is to an install step: a deploy's settings, which no build knows.""" + + def __getattr__(self, name: str) -> NoReturn: + raise AttributeError(f"self.settings.{name}: an install step runs at build, before any deploy gives the app " + "its settings; it reads self.build_settings") + + +def main(argv: list[str]) -> None: + if len(argv) != 2 or ":" not in argv[0]: + raise SystemExit(USAGE) + file, _, name = argv[0].partition(":") + cls = load_class(Path(file), name, ComputerAppEnvironment) + if argv[1] not in {step.name for step in cls.declaration().install_steps}: + raise SystemExit(f"{cls.__name__} has no @install_step method {argv[1]!r}") + app = cls.__new__(cls) + app.settings, app.build_settings = _NoDeploy(), _built_settings(cls, detect().os) + done = getattr(app, argv[1])() + if inspect.isawaitable(done): + asyncio.run(done) + + +if __name__ == "__main__": + main(sys.argv[1:]) diff --git a/packages/agentenv-protocol/tests/probe_fake.py b/packages/agentenv-protocol/tests/probe_fake.py index cd00615b..a6a6e85a 100644 --- a/packages/agentenv-protocol/tests/probe_fake.py +++ b/packages/agentenv-protocol/tests/probe_fake.py @@ -27,7 +27,7 @@ scroll, type_text, ) -from agentenv_protocol.computer import linux +from agentenv_protocol.computer import _linux as linux from agentenv_protocol.computer.conformance.checks import STAGE CLIPBOARD = "agentenv-probe clipboard" diff --git a/packages/agentenv-protocol/tests/test_computer_app_server.py b/packages/agentenv-protocol/tests/test_computer_app_server.py index aaf9c0fe..752afbbd 100644 --- a/packages/agentenv-protocol/tests/test_computer_app_server.py +++ b/packages/agentenv-protocol/tests/test_computer_app_server.py @@ -23,14 +23,16 @@ import agentenv_protocol.computer.launch as launcher # noqa: E402 from agentenv_protocol import client # noqa: E402 -from agentenv_protocol.computer import ComputerAppEnvironment, Launch, ReadyCheck, Ready, ready_check # noqa: E402 -from agentenv_protocol.computer import app_runtime, apps, session # noqa: E402 +from agentenv_protocol.computer import (ComputerAppEnvironment, Launch, ReadyCheck, Ready, linux, # noqa: E402 + ready_check) +from agentenv_protocol.computer import app, app_runtime, apps, session # noqa: E402 from agentenv_protocol.computer.app import COMPUTER_APP, FAILURE, SOCKET # noqa: E402 +from agentenv_protocol.computer.platform import Platform # noqa: E402 +@linux(launch=Launch(command=["mail"], window={"class": "Mail"}, ready=Ready(window=True, timeout_s=30))) class Mail(ComputerAppEnvironment): - launch = Launch(command=["mail"], window={"class": "Mail"}, ready=Ready(window=True, timeout_s=30)) synced = False @ready_check @@ -38,8 +40,9 @@ def inbox_synced(self): return self.synced +@linux(launch=Launch(at="demand", command=["music"], window={"class": "Music"}, ready=Ready(window=True))) class Music(ComputerAppEnvironment): - launch = Launch(at="demand", command=["music"], window={"class": "Music"}, ready=Ready(window=True)) + pass class Sync(ComputerAppEnvironment): @@ -48,6 +51,12 @@ async def caught_up(self): return True +@pytest.fixture(autouse=True) +def on_linux(monkeypatch): + """Every app here runs as on Linux, which its launch is declared for.""" + monkeypatch.setattr(app, "detect", lambda: Platform(os="linux")) + + @pytest.fixture def desktop(monkeypatch, tmp_path): """A session with no windows yet, recording what's launched into it, its launches marked under ``tmp_path``.""" @@ -103,8 +112,8 @@ async def test_checks_that_fail_hang_or_raise_say_so_at_once_without_holding_up_ port = closed.getsockname()[1] exits = [sys.executable, "-c", "import sys; sys.exit('no database yet')"] + @linux(launch=Launch(command=["slow"], ready=Ready(port=port, command=exits))) class Slow(ComputerAppEnvironment): - launch = Launch(command=["slow"], ready=Ready(port=port, command=exits)) @ready_check def stuck(self): diff --git a/packages/agentenv-protocol/tests/test_computer_dataplane.py b/packages/agentenv-protocol/tests/test_computer_dataplane.py index 76108af3..f6a565fc 100644 --- a/packages/agentenv-protocol/tests/test_computer_dataplane.py +++ b/packages/agentenv-protocol/tests/test_computer_dataplane.py @@ -170,6 +170,6 @@ async def test_data_get_uploads_under_a_write_namespace_grant(home): @pytest.mark.asyncio async def test_a_failed_write_leaves_no_temporary_file(home): - with patch("agentenv_protocol.computer.linux.os.replace", side_effect=OSError("disk full")): + with patch("agentenv_protocol.computer._linux.os.replace", side_effect=OSError("disk full")): answer = await _add(FakeDriver(), _part("notes/a.txt")) assert "disk full" in answer["error"]["message"] and list((home / "notes").iterdir()) == [] diff --git a/packages/agentenv-protocol/tests/test_computer_declare.py b/packages/agentenv-protocol/tests/test_computer_declare.py index 9aad8de0..5c057875 100644 --- a/packages/agentenv-protocol/tests/test_computer_declare.py +++ b/packages/agentenv-protocol/tests/test_computer_declare.py @@ -3,6 +3,7 @@ supports, an app's as ComputerAppEnvironment says. An app's code needs only the plain package.""" from __future__ import annotations +import inspect import json import os import re @@ -16,6 +17,11 @@ from agentenv_protocol.computer import AppDeclaration # noqa: E402 from agentenv_protocol.computer._declare import _Stub, _StubModule, declare, main # noqa: E402 +from agentenv_protocol.computer import (APP_SETTINGS_ENV, ComputerAppEnvironment, Launch, Linux, # noqa: E402 + install, linux) +from agentenv_protocol.computer import app as app_module # noqa: E402 +from agentenv_protocol.computer.app import BUILD_SETTINGS # noqa: E402 +from pydantic import BaseModel # noqa: E402 from agentenv_protocol.computer.platform import Platform # noqa: E402 DRIVER = '''import not_installed_here @@ -61,29 +67,58 @@ def press_keys(self, keys): pass from pydantic import BaseModel, SecretStr from agentenv_protocol import add_data, agent_skill, reset_data -from agentenv_protocol.computer import (ComputerAppEnvironment, Launch, Linux, Ready, Service, computer_app, - install_step, ready_check) +from agentenv_protocol.computer import (AptSource, ComputerAppEnvironment, Download, Launch, Packages, Ready, + RunScript, Service, computer_app, install, install_step, linux, ready_check) + + +class FreeBSD(BaseModel): + ports: list[str] = [] + + +class LinuxSettings(BaseModel): + font: str = "DejaVu Sans" + + +class LinuxBuildSettings(BaseModel): + fonts: list[str] = [] + + +class LeakyBuildSettings(BaseModel): + key: SecretStr = SecretStr("") @computer_app(supports={supports}, permissions=["camera"]) +@linux(files="linux", python="requirements.txt", packages=Packages(apt=["dovecot-imapd"]), + sources=[AptSource(apt="https://ppa.example/ubuntu", suite="noble", key="ppa.asc")], + downloads=[Download(url="https://example.com/mbox", sha256="{sha}", to="data/mbox", arch="arm64")], + run=[RunScript(script="setup.sh"), RunScript(script="profile.sh", scope="session")], + services=[Service(name="imap", command="bin/imap")], ports=[143], + launch=Launch(at="demand", command=["thunderbird"], window={{"class": "Thunderbird"}}, + ready=Ready(window=True, timeout_s=30)), + settings=LinuxSettings, build_settings=LinuxBuildSettings) +@install("freebsd", FreeBSD(ports=["mail/thunderbird"]), launch=Launch(command=["thunderbird"])) class Mail(ComputerAppEnvironment): - linux = Linux(files="linux", python="requirements.txt", services=[Service(name="imap", command="bin/imap")]) - launch = Launch(at="demand", command=["thunderbird"], window={{"class": "Thunderbird"}}, - ready=Ready(window=True, timeout_s=30)) - class Settings(BaseModel): account: str = "user@example.com" password: SecretStr = SecretStr("") + class BuildSettings(BaseModel): + locale: str = "en_US" + writeOnly: bool = False + flags: dict = {{"writeOnly": True}} + @install_step() def packages(self): not_installed_here.apt() - @install_step(scope="session") + @install_step(scope="session", os="linux") def profile(self): pass @ready_check def inbox_open(self): return True + @ready_check(os=["freebsd"]) + def ports_built(self): return True + @agent_skill def guide(self): return "Mail", "Read mail." @@ -165,26 +200,44 @@ def test_main_prints_only_the_declaration_or_exits_1_with_why_it_cant(folder, ca def test_an_app_class_whose_imports_arent_installed_declares_what_it_is_to_a_computer(folder): - (folder / "app.py").write_text(APP.format(supports=LINUX)) + (folder / "app.py").write_text(APP.format(supports=LINUX, sha='a' * 64)) declared = declare(folder, "app.py", "Mail") assert isinstance(declared, AppDeclaration) settings = declared.settings - assert declared.model_dump(mode="json", exclude_none=True, exclude={"settings"}) == { + schemas = {"settings", "build_settings"} + assert declared.model_dump(mode="json", exclude_none=True, exclude={**dict.fromkeys(schemas, True), "os": { + "__all__": schemas}}) == { "supports": LINUX, "permissions": ["camera"], - "install": {"linux": {"files": "linux", "python": "requirements.txt", - "services": [{"name": "imap", "command": "bin/imap"}]}}, - "launch": {"at": "demand", "command": ["thunderbird"], "window": {"class": "Thunderbird"}, - "ready": {"window": True, "timeout_s": 30.0}}, + "os": { + "linux": { + "install": { + "files": "linux", "packages": {"apt": ["dovecot-imapd"]}, "python": "requirements.txt", + "sources": [{"apt": "https://ppa.example/ubuntu", "suite": "noble", "components": [], + "key": "ppa.asc"}], + "downloads": [{"url": "https://example.com/mbox", "sha256": "a" * 64, "to": "data/mbox", + "arch": "arm64"}], + "run": [{"script": "setup.sh", "scope": "system"}, {"script": "profile.sh", "scope": "session"}], + "services": [{"name": "imap", "command": "bin/imap"}], "ports": [143]}, + "launch": {"at": "demand", "command": ["thunderbird"], "window": {"class": "Thunderbird"}, + "ready": {"window": True, "timeout_s": 30.0}}}, + "freebsd": {"install": {"ports": ["mail/thunderbird"]}, + "launch": {"at": "boot", "command": ["thunderbird"]}}}, "skill": {"method": "guide"}, "data": ["data/add", "data/reset"], - "install_steps": [{"name": "packages", "scope": "system"}, {"name": "profile", "scope": "session"}], - "ready_checks": ["inbox_open"]} + "install_steps": [{"name": "packages", "scope": "system"}, + {"name": "profile", "scope": "session", "os": ["linux"]}], + "ready_checks": [{"name": "inbox_open"}, {"name": "ports_built", "os": ["freebsd"]}]} assert list(settings["properties"]) == ["account", "password"] and settings["properties"]["password"]["writeOnly"] + assert list(declared.build_settings["properties"]) == ["locale", "writeOnly", "flags"] + linux = declared.os["linux"] + assert set(linux.settings["properties"]) == {"account", "password", "font"} + assert set(linux.build_settings["properties"]) == {"locale", "writeOnly", "flags", "fonts"} + assert (declared.os["freebsd"].settings, declared.os["freebsd"].build_settings) == (None, None) def test_an_app_declaring_its_skill_twice_is_refused(folder): - (folder / "app.py").write_text(APP.format(supports=LINUX)) + (folder / "app.py").write_text(APP.format(supports=LINUX, sha='a' * 64)) with pytest.raises(ValueError, match=re.escape("Twice declares its skill twice: skill = 'skill.md' and " "@agent_skill guide")): @@ -192,36 +245,104 @@ def test_an_app_declaring_its_skill_twice_is_refused(folder): @pytest.mark.parametrize("change, problem", [ - ('launch = Launch(command=["mail"], ready=Ready(timeout_s=5))', "Ready waits on nothing: set window, port or command"), - ('launch = Launch(command=["mail"], ready=Ready(window=True))', "ready waits on a window: name it with Launch(window="), - ('launch = Launch(command=["mail"], window={"role": "main"})', "window matches by class, instance, title, not"), -]) -def test_an_app_whose_launch_or_readiness_cant_be_checked_is_refused_saying_why(folder, capsys, change, problem): - (folder / "app.py").write_text(APP.format(supports=LINUX) + f""" + ('@linux(launch=Launch(command=["mail"], ready=Ready(timeout_s=5)))', + "Ready waits on nothing: set window, port or command"), + ('@linux(launch=Launch(command=["mail"], ready=Ready(window=True)))', + "ready waits on a window: name it with Launch(window="), + ('@linux(launch=Launch(command=["mail"], window={"role": "main"}))', + "window matches by class, instance, title, not"), + ('@linux(downloads=[Download(url="https://example.com/t", sha256="abc", to="t")])', + "Download.sha256: String should match pattern '^[0-9a-f]{64}$'"), + ('@linux(run=[RunScript(script="../setup.sh")])', + "RunScript.script: '../setup.sh' must be a path inside the layer's folder"), + ('@linux(skill="skill-linux.md")', + "Changed declares its skill twice on linux: skill = 'skill-linux.md' there and @agent_skill guide, which its " + "server serves on every OS"), + ('@linux(build_settings=LeakyBuildSettings)', + "LeakyBuildSettings holds a secret at properties.key, which its computer's image would keep"), + ('@install_step(os="windows")\n def fonts(self): pass', + "Changed.fonts runs on 'windows', which Changed declares no install for; it declares freebsd, linux"), + ('@ready_check(os="macos")\n def window_shown(self): return True', + "Changed.window_shown runs on 'macos', which Changed declares no install for"), + ('class BuildSettings(BaseModel): key: SecretStr = SecretStr("")', + "Changed.BuildSettings holds a secret at properties.key, which its computer's image would keep; declare it in " + "Settings"), + ('class BuildSettings(BaseModel): keys: list[SecretStr] = []', + "Changed.BuildSettings holds a secret at properties.keys.items, which"), + ('class BuildSettings(BaseModel): mail: Mail.Settings = Mail.Settings()', + "Changed.BuildSettings holds a secret at $defs.Settings.properties.password, which"), +], ids=["no-wait", "no-window", "window-key", "download-sha", "script-escape", "os-skill-twice", "os-build-secret", + "step-os", + "check-os", "build-secret", "build-secrets-listed", "build-secret-nested"]) +def test_an_app_whose_declaration_cant_be_checked_is_refused_saying_why(folder, capsys, change, problem): + decorator, body = (change, "pass") if change.startswith("@linux") else ("", change) + (folder / "app.py").write_text(APP.format(supports=LINUX, sha='a' * 64) + f""" @computer_app(supports={LINUX}) +{decorator} class Changed(Mail): - {change} + {body} """) assert main([str(folder), "app.py", "Changed"]) == 1 assert problem in capsys.readouterr().err.splitlines()[-1] -def test_an_app_reads_the_settings_its_computer_gives_it_by_its_name(monkeypatch): - from pydantic import BaseModel +@pytest.mark.parametrize("os_type", ["linux", "macos"]) +def test_an_app_reads_its_settings_build_settings_and_launch_as_declared_for_the_os_it_runs_on(monkeypatch, tmp_path, + os_type): + class LinuxSettings(BaseModel): + font: str = "DejaVu Sans" - from agentenv_protocol.computer import APP_SETTINGS_ENV, ComputerAppEnvironment + class LinuxBuildSettings(BaseModel): + edition: str = "linux" + @install("linux", Linux(), launch=Launch(command=["notes"]), settings=LinuxSettings, + build_settings=LinuxBuildSettings) class Notes(ComputerAppEnvironment): class Settings(BaseModel): signature: str = "" lines: int = 1 + class BuildSettings(BaseModel): + edition: str = "home" + locale: str = "en_US" + + monkeypatch.setattr(app_module, "detect", lambda: Platform(os=os_type)) monkeypatch.setenv("ENVIRONMENT_NAME", "notes") monkeypatch.setenv(APP_SETTINGS_ENV, json.dumps({"notes": {"signature": "-- Ada"}, "mail": {"lines": 9}})) + monkeypatch.chdir(tmp_path) + assert Notes().build_settings.edition == ("linux" if os_type == "linux" else "home") + (tmp_path / BUILD_SETTINGS).parent.mkdir() + (tmp_path / BUILD_SETTINGS).write_text(json.dumps({"locale": "fr_FR"})) + + notes = Notes() + + assert (notes.settings.signature, notes.settings.lines, notes.build_settings.locale) == ("-- Ada", 1, "fr_FR") + if os_type == "linux": + assert (notes.settings.font, notes.build_settings.edition, notes.launch.command) == ( + "DejaVu Sans", "linux", ["notes"]) + else: + assert (type(notes.settings), notes.build_settings.edition, notes.launch) == (Notes.Settings, "home", None) + + +def test_a_declaration_stored_before_each_os_had_its_own_part_reads_as_its_linux_part(): + stored = {"supports": LINUX, "permissions": [], "settings": {"type": "object"}, "data": [], "install_steps": [], + "install": {"linux": {"files": ".", "services": [{"name": "imap", "command": "bin/imap"}]}}, + "launch": {"at": "demand", "command": ["mail"]}, "ready_checks": ["inbox_open"]} + + declared = AppDeclaration.model_validate(stored) + + assert declared.os["linux"].install == stored["install"]["linux"] + assert declared.os["linux"].launch == Launch(at="demand", command=["mail"]) + assert [check.model_dump(exclude_none=True) for check in declared.ready_checks] == [{"name": "inbox_open"}] + assert AppDeclaration.model_validate(declared.model_dump(mode="json")) == declared + + +def test_linux_takes_every_field_of_a_linux_install_and_what_install_adds_for_an_os(): + added = set(inspect.signature(install).parameters) - {"os_type", "spec"} - assert Notes().settings == Notes.Settings(signature="-- Ada", lines=1) + assert set(inspect.signature(linux).parameters) == set(Linux.model_fields) | added def test_an_apps_code_needs_only_the_plain_package(): diff --git a/packages/agentenv-protocol/tests/test_computer_extensions.py b/packages/agentenv-protocol/tests/test_computer_extensions.py index cbd0a9c8..639640a6 100644 --- a/packages/agentenv-protocol/tests/test_computer_extensions.py +++ b/packages/agentenv-protocol/tests/test_computer_extensions.py @@ -13,7 +13,7 @@ from computer_fake import FakeDriver, MinimalDriver, http_client # noqa: E402 -from agentenv_protocol.computer import ComputerSettings, ExecResult, linux, run_in_session, set_display # noqa: E402 +from agentenv_protocol.computer import ComputerSettings, ExecResult, _linux, run_in_session, set_display # noqa: E402 pytestmark = pytest.mark.skipif(sys.platform == "win32", reason="the Linux defaults need a POSIX host") @@ -159,14 +159,14 @@ async def run(self, argv, cwd, env, timeout_s): @pytest.mark.asyncio async def test_a_background_apps_output_is_drained_after_the_call_returns(monkeypatch): - monkeypatch.setattr(linux, "DRAIN_S", 0.1) + monkeypatch.setattr(_linux, "DRAIN_S", 0.1) async with http_client(FakeDriver()) as http: r = await http.post("/agentenv/ext/computer-exec/run", json=_run( argv=["sh", "-c", "echo now; (sleep 0.4; echo later) &"])) - draining = len(linux._draining) + draining = len(_linux._draining) async def drained(): - while linux._draining: + while _linux._draining: await asyncio.sleep(0.05) await asyncio.wait_for(drained(), timeout=10) diff --git a/packages/agentenv-protocol/tests/test_computer_serve.py b/packages/agentenv-protocol/tests/test_computer_serve.py index 8c7662cc..84aa7711 100644 --- a/packages/agentenv-protocol/tests/test_computer_serve.py +++ b/packages/agentenv-protocol/tests/test_computer_serve.py @@ -11,8 +11,8 @@ pytest.importorskip("PIL") pytest.importorskip("uvicorn") -from agentenv_protocol.computer import ComputerDriverEnvironment, serve_app # noqa: E402 -from agentenv_protocol.computer.app import FAILURE, SOCKET # noqa: E402 +from agentenv_protocol.computer import ComputerDriverEnvironment, run_install_step, serve_app # noqa: E402 +from agentenv_protocol.computer.app import BUILD_SETTINGS, FAILURE, SOCKET # noqa: E402 from agentenv_protocol.computer.app_runtime import AppRuntime # noqa: E402 from agentenv_protocol.computer.serve import USAGE, main # noqa: E402 @@ -69,6 +69,41 @@ def __init__(self): ''' +STEPS = '''from pathlib import Path + +from pydantic import BaseModel + +from agentenv_protocol.computer import ComputerAppEnvironment, install_step + + +class Notes(ComputerAppEnvironment): + class Settings(BaseModel): + token: str + + class BuildSettings(BaseModel): + folder: str = "notes" + edition: str + + def __init__(self): + raise RuntimeError("no deploy's settings at build") + + @install_step() + def make_folder(self): + Path(self.build_settings.folder).mkdir() + + @install_step(scope="session") + async def write_index(self): + Path(self.build_settings.folder, "index").write_text(self.build_settings.edition) + + @install_step() + def read_token(self): + Path("token").write_text(self.settings.token) + + def helper(self): + pass +''' + + @pytest.fixture def isolated(monkeypatch): """Imports that a class's file makes, undone after the test.""" @@ -178,3 +213,23 @@ def test_an_apps_server_that_cant_start_says_why_in_the_apps_folder(tmp_path, mo serve_app.main(argv) assert (tmp_path / FAILURE).read_text() == failure == f"{type(stopped.value).__name__}: {stopped.value}" + + +def test_an_apps_install_step_runs_without_its_init_on_the_build_settings_its_env_fixed(tmp_path, monkeypatch, + isolated): + (tmp_path / "app.py").write_text(STEPS) + (tmp_path / BUILD_SETTINGS).parent.mkdir() + (tmp_path / BUILD_SETTINGS).write_text('{"edition": "pro"}') + monkeypatch.chdir(tmp_path) + + run_install_step.main(["app.py:Notes", "make_folder"]) + run_install_step.main(["app.py:Notes", "write_index"]) + + assert (tmp_path / "notes" / "index").read_text() == "pro" + with pytest.raises(AttributeError, match="self.settings.token: an install step runs at build, before any deploy " + "gives the app its settings; it reads self.build_settings"): + run_install_step.main(["app.py:Notes", "read_token"]) + with pytest.raises(SystemExit, match="Notes has no @install_step method 'helper'"): + run_install_step.main(["app.py:Notes", "helper"]) + with pytest.raises(SystemExit, match="usage: "): + run_install_step.main(["app.py:Notes"]) diff --git a/src/agent_env/drivers/README.md b/src/agent_env/drivers/README.md index 4be6d407..893bcc80 100644 --- a/src/agent_env/drivers/README.md +++ b/src/agent_env/drivers/README.md @@ -9,8 +9,8 @@ alias `aliases.toml` gives it, or by its id, `@local/agentenv-framework/drivers/ A run that names it writes it: its files, what its class declares for each platform it supports, read without `mss`, `python-xlib` or PyAV installed, and how a computer installs it. Its `env.toml`'s `[install.linux]` has a -computer env's composer put the folder in `/opt/agentenv/drivers/reference/`, install `requirements.txt` into a venv -there, and run `agentenv_protocol.computer.serve driver.py:ReferenceDriver` as a session service on `port`, 18765. It deploys only inside a computer env, as +computer env's composer put the folder in `/opt/agentenv/drivers/reference/`, install `requirements.txt`, resolved and +fetched once at put, into a venv there, and run `agentenv_protocol.computer.serve driver.py:ReferenceDriver` as a session service on `port`, 18765. It deploys only inside a computer env, as the shipped `desktop` example's driver. `python -m agentenv_protocol.computer.conformance ` checks it, or any driver, where it runs. diff --git a/src/agent_env/env/envs/computer/app_env.py b/src/agent_env/env/envs/computer/app_env.py index ed3e392c..87bc66bb 100644 --- a/src/agent_env/env/envs/computer/app_env.py +++ b/src/agent_env/env/envs/computer/app_env.py @@ -1,13 +1,14 @@ -"""Computer app env: software on a computer, a ``ComputerAppEnvironment`` subclass, with its files and what it declares, -read from the class at put. Deployed alone it is a one-app computer env: the computer and driver its env.toml names, -composed with the app installed under its prefix, its card named after it.""" +"""Computer app env: software on a computer, a ``ComputerAppEnvironment`` subclass, with its files, what it declares, +read from the class at put, and the values of its build settings the env fixes in the image. Deployed alone it is a +one-app computer env: the computer and driver its env.toml names, composed with the app installed under its prefix, its +card named after it.""" from __future__ import annotations from pathlib import PurePosixPath from typing import TYPE_CHECKING, Any, ClassVar, Optional, Self -from agentenv_protocol.computer import AppDeclaration +from agentenv_protocol.computer import AppDeclaration, Launch from agent_env.artifact import Artifact from agent_env.artifact.artifacts.computer import ComputerArtifact @@ -15,7 +16,7 @@ from agent_env.env.env import Env from agent_env.env.envs.computer.utils import composer, gate from agent_env.env.envs.computer.env import ComputerEnv -from agent_env.env.envs.computer.utils.install import AppInstall, installs, missing +from agent_env.env.envs.computer.utils.install import AppInstall, folder_problems, installs from agent_env.env.envs.computer.driver_env import ComputerDriverEnv, declared from agent_env.env.store import get_env_store from agent_env.store.ids import derive_id @@ -33,11 +34,13 @@ class ComputerAppEnv(ComputerEnv): def __init__(self, id: str, version: Optional[int], environment_name: str, module: str, files: FileArtifactUniverse, declaration: AppDeclaration, computer: Optional[ComputerArtifact] = None, - driver: Optional[ComputerDriverEnv] = None, metadata: Optional[dict[str, Any]] = None): + driver: Optional[ComputerDriverEnv] = None, build_settings: Optional[dict[str, Any]] = None, + metadata: Optional[dict[str, Any]] = None): super().__init__(id, version, environment_name, computer, driver, metadata=metadata) self.module = module self.files = files self.declaration = declaration + self.build_settings = build_settings or {} @property def install(self) -> dict[str, AppInstall]: @@ -46,12 +49,30 @@ def install(self) -> dict[str, AppInstall]: def composed_apps(self) -> list[ComputerAppEnv]: return [self] - def secret_settings(self) -> set[str]: - """The names of its settings that are secrets: a ``SecretStr``, optional or not, which its schema marks - ``writeOnly``.""" - return {name for name, schema in self.declaration.settings.get("properties", {}).items() + def settings_schema(self, os_type: str | None) -> dict[str, Any]: + """The JSON schema of its settings on ``os_type``: its class's, with the fields that OS adds; with no OS type, + its class's.""" + on = self.declaration.os.get(os_type) + own = on.settings if on is not None else None + return own if own is not None else self.declaration.settings + + def build_settings_schema(self, os_type: str | None) -> dict[str, Any] | None: + """The JSON schema of its build settings on ``os_type``, as ``settings_schema``; None when it declares none.""" + on = self.declaration.os.get(os_type) + own = on.build_settings if on is not None else None + return own if own is not None else self.declaration.build_settings + + def secret_settings(self, os_type: str | None) -> set[str]: + """The names of its settings on ``os_type`` that are secrets: a ``SecretStr``, optional or not, which its schema + marks ``writeOnly``.""" + return {name for name, schema in self.settings_schema(os_type).get("properties", {}).items() if any(option.get("writeOnly") for option in [schema, *schema.get("anyOf", [])])} + def launch(self, os_type: str) -> Launch | None: + """How it launches on ``os_type``.""" + on = self.declaration.os.get(os_type) + return on.launch if on is not None else None + def to_dict(self) -> dict: return {**Env.to_dict(self), "service_name": self.environment_name, "environment_name": self.environment_name, # dual-write with service_name during the rename @@ -59,7 +80,8 @@ def to_dict(self) -> dict: "files": {"id": self.files.id, "version": self.files.version, "type": self.files.type}, "declaration": self.declaration.model_dump(mode="json", exclude_none=True), "computer": {"id": self.computer.id, "version": self.computer.version} if self.computer else None, - "driver": {"id": self.driver.id, "version": self.driver.version} if self.driver else None} + "driver": {"id": self.driver.id, "version": self.driver.version} if self.driver else None, + **({"build_settings": self.build_settings} if self.build_settings else {})} @classmethod def from_dict(cls, data: dict) -> ComputerAppEnv: @@ -70,20 +92,22 @@ def from_dict(cls, data: dict) -> ComputerAppEnv: return cls(id=data["id"], version=data.get("version"), environment_name=data["environment_name"], module=data["module"], files=Artifact.get(data["files"]["id"], version=data["files"]["version"]), declaration=AppDeclaration.model_validate(data["declaration"]), computer=computer, driver=driver, - metadata=data.get("metadata") or {}) + build_settings=data.get("build_settings"), metadata=data.get("metadata") or {}) @classmethod def put(cls, *, id: str, environment_name: str, module: str, files: FileArtifactUniverse, declaration: AppDeclaration, computer: Optional[ComputerArtifact] = None, - driver: Optional[ComputerDriverEnv] = None, compose: bool = True, - on_build: Optional[Callable[[], None]] = None) -> Self: - """Write the app once what it declares checks out, and with a computer and a driver, once it fits them (gate - 1); then compose it with them unless ``compose`` is False. Refused, naming every problem, before anything is - stored.""" + driver: Optional[ComputerDriverEnv] = None, build_settings: Optional[dict[str, Any]] = None, + compose: bool = True, on_build: Optional[Callable[[], None]] = None) -> Self: + """Write the app once what it declares and ``build_settings``, its build settings' values, check out, and with + a computer and a driver, once it fits them (gate 1); then compose it with them unless ``compose`` is False. + Refused, naming every problem, before anything is stored.""" app = cls(id=id, version=None, environment_name=environment_name, module=module, files=files, - declaration=declaration, computer=computer, driver=driver) + declaration=declaration, computer=computer, driver=driver, build_settings=build_settings) _, problems = _installs(declaration) problems += _class_beyond_reach(module, declaration) + problems += gate.check_app_build_settings(app, app.build_settings, "build_settings", + computer.os.type if computer is not None else None) if (computer is None) != (driver is None): problems.append("a computer app deploys alone on a computer and a driver; name both or neither") elif computer is not None and not problems: @@ -99,15 +123,18 @@ def put(cls, *, id: str, environment_name: str, module: str, files: FileArtifact @classmethod def from_toml(cls, data: dict, ctx: AuthoringContext) -> ComputerAppEnv: """Write the app an env.toml declares, its class read in a process of its own (``declared``), its files stored - as a file universe, with the computer and driver env.toml names; its environment_name the folder's when it sets - none.""" + as a file universe, with the computer and driver and the build settings' values env.toml names; its + environment_name the folder's when it sets none.""" fields = cls.accept_toml(data, ctx) declaration = AppDeclaration.model_validate(declared(ctx, fields["module"], cls.type)) files = ctx.files() found, problems = _installs(declaration) - problems += [problem for os_type, install in found.items() for problem in missing(os_type, install, files)] - if (skill := declaration.skill and declaration.skill.file) is not None and skill not in files: - problems.append(f"skill: {skill!r} isn't in this folder") + problems += [problem for os_type, install in found.items() + for problem in folder_problems(os_type, install, files)] + skills = [("skill", declaration.skill), + *((f"{os_type}.skill", on.skill) for os_type, on in declaration.os.items())] + problems += [f"{where}: {skill.file!r} isn't in this folder" for where, skill in skills + if skill is not None and skill.file is not None and skill.file not in files] if problems: ctx.refuse(list(map(ctx.config_problem, problems))) computer = ctx.artifact(fields["computer"], ComputerArtifact) if "computer" in fields else None @@ -116,16 +143,17 @@ def from_toml(cls, data: dict, ctx: AuthoringContext) -> ComputerAppEnv: try: return cls.put(id=ctx.id, environment_name=fields.get("environment_name", ctx.name), module=fields["module"], files=universe, declaration=declaration, computer=computer, - driver=driver, on_build=ctx.on_build) + driver=driver, build_settings=fields.get("build_settings"), on_build=ctx.on_build) except ValueError as e: ctx.refuse([str(e)]) @classmethod def accept_toml(cls, data: dict, ctx: AuthoringContext) -> dict: """The keys of ``data``, an env.toml, a computer app env takes: ``module = ":"``, naming a file of - its folder; ``environment_name``; and the ``computer`` and ``driver`` it deploys alone on. Raises BundleError - listing every problem.""" - fields, refused = ctx.accepted(data, environment_name=str, module=str, computer=object, driver=object) + its folder; ``environment_name``; the ``computer`` and ``driver`` it deploys alone on; and ``build_settings``, + the values of its build settings. Raises BundleError listing every problem.""" + fields, refused = ctx.accepted(data, environment_name=str, module=str, computer=object, driver=object, + build_settings=dict) files, module, problems = ctx.files(), fields.get("module"), [] if module is None and "module" not in data: problems.append('module is required: the app\'s file and class, such as "app.py:MailApp"') @@ -146,12 +174,13 @@ def deploy_refusal(self, *, sandbox_type: str | None = None, env_params: Optiona def _installs(declaration: AppDeclaration) -> tuple[dict[str, AppInstall], list[str]]: """Its installs by OS type, each checked as the composer installs it, and the problems with them.""" - return installs({os_type: linux.model_dump() for os_type, linux in declaration.install.items()}, AppInstall) + return installs({os_type: on.install for os_type, on in declaration.os.items()}, AppInstall) def _class_beyond_reach(module: str, declaration: AppDeclaration) -> list[str]: """An install that leaves out the class's file, which the app's own server serves from the app's folder.""" file = PurePosixPath(module.partition(":")[0]) + folders = {os_type: on.install.get("files", ".") for os_type, on in declaration.os.items()} return [f"install.{os_type}.files: the app's server serves its class from the app's folder, which needs " - f"{str(file)!r} in {linux.files!r}" for os_type, linux in declaration.install.items() - if linux.files != "." and not file.is_relative_to(linux.files)] + f"{str(file)!r} in {folder!r}" for os_type, folder in folders.items() + if folder != "." and not file.is_relative_to(folder)] diff --git a/src/agent_env/env/envs/computer/driver_env.py b/src/agent_env/env/envs/computer/driver_env.py index 08b4a73b..6896fe40 100644 --- a/src/agent_env/env/envs/computer/driver_env.py +++ b/src/agent_env/env/envs/computer/driver_env.py @@ -14,7 +14,7 @@ from agent_env.artifact import Artifact from agent_env.artifact.artifacts.file_artifact_universe import FileArtifactUniverse from agent_env.env.env import Env -from agent_env.env.envs.computer.utils.install import Install, installs, missing +from agent_env.env.envs.computer.utils.install import Install, folder_problems, installs from agent_env.store.ids import derive_id if TYPE_CHECKING: @@ -83,7 +83,7 @@ def accept_toml(cls, data: dict, ctx: AuthoringContext) -> dict: problems.append(f"port must be 1 to {MAX_PORT}, not {fields['port']}") fields["install"], invalid = installs(fields.get("install", {})) problems += invalid + [problem for os_type, install in fields["install"].items() - for problem in missing(os_type, install, files)] + for problem in folder_problems(os_type, install, files)] if refused or problems: ctx.refuse([*refused, *map(ctx.config_problem, problems)]) return fields diff --git a/src/agent_env/env/envs/computer/utils/boot.py b/src/agent_env/env/envs/computer/utils/boot.py index f348702c..5e5b2bbe 100644 --- a/src/agent_env/env/envs/computer/utils/boot.py +++ b/src/agent_env/env/envs/computer/utils/boot.py @@ -68,14 +68,16 @@ def settings_environment(computer: ComputerArtifact, values: dict[str, Any], *, return environment -def app_settings_environment(apps: list[ComputerAppEnv], app_settings: dict[str, dict]) -> dict[str, str]: - """The environment that hands each of ``apps`` the values ``app_settings`` gives it, by its name, in - ``APP_SETTINGS_ENV``, a secret's resolved through the secret store; nothing when none has any.""" +def app_settings_environment(apps: list[ComputerAppEnv], app_settings: dict[str, dict], + os_type: str) -> dict[str, str]: + """The environment that hands each of ``apps``, on a computer of ``os_type``, the values ``app_settings`` gives it, + by its name, in ``APP_SETTINGS_ENV``, a secret's resolved through the secret store; nothing when none has any.""" given = [(app, app_settings[app.environment_name]) for app in apps if app_settings.get(app.environment_name)] if not given: return {} return {APP_SETTINGS_ENV: json.dumps({app.environment_name: { - name: get_config().resolve_reference(value) if name in app.secret_settings() and value is not None else value + name: get_config().resolve_reference(value) if name in app.secret_settings(os_type) and value is not None + else value for name, value in values.items()} for app, values in given})} @@ -115,11 +117,11 @@ async def discard(sandbox: Sandbox) -> None: await asyncio.to_thread(remove_local_work_dir, sandbox.sandbox_id) -def ready_timeout(apps: list[ComputerAppEnv]) -> float: - """How long a computer of ``apps`` may take to be ready: ``READY_TIMEOUT_S``, or longer when an app that starts with - it gives itself longer.""" +def ready_timeout(apps: list[ComputerAppEnv], os_type: str) -> float: + """How long a computer of ``os_type`` and ``apps`` may take to be ready: ``READY_TIMEOUT_S``, or longer when an app + that starts with it there gives itself longer.""" return max([READY_TIMEOUT_S, *(launch.ready.timeout_s + APP_READY_MARGIN_S for app in apps - if (launch := app.declaration.launch) and launch.at == "boot" and launch.ready)]) + if (launch := app.launch(os_type)) and launch.at == "boot" and launch.ready)]) async def ready(url: str, timeout: float) -> dict: diff --git a/src/agent_env/env/envs/computer/utils/composer.py b/src/agent_env/env/envs/computer/utils/composer.py index bfdf61fd..d8c516be 100644 --- a/src/agent_env/env/envs/computer/utils/composer.py +++ b/src/agent_env/env/envs/computer/utils/composer.py @@ -29,7 +29,7 @@ from typing import TYPE_CHECKING, Any from agentenv_protocol.computer import apps as installed_apps -from agentenv_protocol.computer.app import SOCKET +from agentenv_protocol.computer.app import BUILD_SETTINGS, SOCKET from agentenv_protocol.computer.conformance import run_probe from agent_env.artifact.artifacts.computer import ComputerArtifact @@ -39,7 +39,7 @@ from agent_env.config import get_config from agent_env.entity.export import Unwritable, checked_path from agent_env.env.env import Env -from agent_env.env.envs.computer.utils import boot, gate +from agent_env.env.envs.computer.utils import boot, gate, packages from agent_env.env.envs.computer.utils.install import Install from agent_env.providers.sandbox_providers.host_images import build_for, host_platform, platform_tag from agent_env.store.base import ConcurrentModificationError @@ -67,10 +67,6 @@ KEY_LABEL = "dev.agentenv.computer-key" # What a build installed and whether it was emulated, kept on its image for an env that reuses it. INSTALLED_LABEL, EMULATED_LABEL = "dev.agentenv.installed", "dev.agentenv.emulated" -# What a file starts with when it's only ever run: a script or a Linux binary, made executable in the image, since the -# store keeps no file modes. -_RUN_ONLY = (b"#!", b"\x7fELF") -_EXECUTABLE = 0o755 # A pair's statuses once its image is in the store. _BUILT = ("unverified", "ready") _WRITE_TRIES = 4 @@ -172,7 +168,7 @@ async def _probe(links: list[SandboxProvider], env: ComputerEnv, driver: Compute try: booted = await boot.boot(link, image=image, port=driver.port, environment=environment, network_policy=None, timeout=_PROBE_TTL_S, attribution=None, - ready_s=boot.ready_timeout(env.composed_apps())) + ready_s=boot.ready_timeout(env.composed_apps(), env.computer.os.type)) except Exception as e: problems.append(f"{type(link).__name__}: {e}") continue @@ -191,14 +187,34 @@ def _built(env: ComputerEnv, base: ComputerArtifact, driver: ComputerDriverEnv, stage, key = "build", None try: with tempfile.TemporaryDirectory(prefix="agentenv-compose-") as work: - context = Path(work) + context, fetched = Path(work) / "context", {} + context.mkdir() source, base_identity = _base(base, arch) layer = _stage_driver(context, driver, base) apps = [_stage_app(context, app, base) for app in env.composed_apps()] - dockerfile = _dockerfile(source, [layer, *apps], base, _build_settings(env, base), _stage_protocol(context)) + layers = [layer, *apps] + wants = _wants(layers, _stage_protocol(context)) + platform = f"linux/{arch}" + emulated = platform != host_platform() + unemulated = f"this machine can't build {platform}: its Docker has no emulation for it" if ( + emulated and platform not in _emulated_platforms()) else None + ready = _once(lambda: _ready_base(base, arch)) + inputs = packages.inputs_digest(wants, context, base_identity, arch) + lock = packages.stored(inputs, env.stored_id) if inputs is not None else None + if inputs is not None and lock is None: + if unemulated: + return _failed(env, form, arch, "resolve", None, unemulated) + stage = "resolve" + building() + ready() + lock, fetched = packages.resolve(inputs, wants, context, source, arch, env.stored_id, + Path(work) / "resolve") + stage = "build" + dockerfile = _dockerfile(source, layers, base, _build_settings(env, base), lock) key = "sha256:" + hashlib.sha256(json.dumps({ - "base": base_identity, "dockerfile": dockerfile, "context": _digest(context), "form": form, - "arch": arch, "composer": _composer_digest(), + "base": base_identity, "dockerfile": dockerfile, "context": _digest(context), + "lock": lock.digest() if lock is not None else None, "form": form, "arch": arch, + "composer": _composer_digest(), }, sort_keys=True).encode()).hexdigest() store = get_config().get_image_store_for(env.stored_id) store.ensure_repository(REPOSITORY) @@ -206,15 +222,14 @@ def _built(env: ComputerEnv, base: ComputerArtifact, driver: ComputerDriverEnv, if (pinned := _served(ref, store)) is not None: kept = labels(pinned, arch, store.auth(pinned)) return {"status": "unverified", "key": key, "image": pinned, "reused": True, - "emulated": kept.get(EMULATED_LABEL) == "true", + "emulated": kept.get(EMULATED_LABEL) == "true", "lock": inputs, "installed": json.loads(kept.get(INSTALLED_LABEL, "{}")), "built_at": _now()} - platform = f"linux/{arch}" - emulated = platform != host_platform() - if emulated and platform not in _emulated_platforms(): - return _failed(env, form, arch, "build", key, f"this machine can't build {platform}: its Docker has " - "no emulation for it") + if unemulated: + return _failed(env, form, arch, "build", key, unemulated) building() - _ready_base(base, arch) + ready() + if lock is not None: + packages.stage(lock, context, env.stored_id, fetched) (context / "Dockerfile").write_text(f"{dockerfile}LABEL {KEY_LABEL}={key}\n") local = f"agentenv-computer:{key[7:23]}-{arch}" build_image(context / "Dockerfile", context, local, platform=platform, labels={KEY_LABEL: key}) @@ -224,7 +239,9 @@ def _built(env: ComputerEnv, base: ComputerArtifact, driver: ComputerDriverEnv, stage = "push" push_local_image(local, ref, store) return {"status": "unverified", "key": key, "image": pin_digest(ref, store.auth(ref)), "reused": False, - "emulated": emulated, "installed": installed, "built_at": _now()} + "emulated": emulated, "lock": inputs, "installed": installed, "built_at": _now()} + except packages.ResolveError as e: + return _failed(env, form, arch, "resolve", None, str(e), log=e.log or None) except DockerBuildError as e: # Docker can list a platform it has lost the emulation for, until it restarts; the build says so. unemulated = "exec format error" in str(e) @@ -281,6 +298,8 @@ class _Layer: install: Install conf: str app: bool = False + served: str | None = None + steps: tuple[tuple[str, str], ...] = () def _stage_driver(context: Path, driver: ComputerDriverEnv, base: ComputerArtifact) -> _Layer: @@ -293,16 +312,21 @@ def _stage_driver(context: Path, driver: ComputerDriverEnv, base: ComputerArtifa def _stage_app(context: Path, app: ComputerAppEnv, base: ComputerArtifact) -> _Layer: - """Write the app's install files under ``context/apps//`` and its supervisord programs beside them, run under - its name: its services and its own server, which serves its class from its venv and launches the app. Its folder - under ``APPS`` is named for its id's last part.""" + """Write the app's install files under ``context/apps//``, with the values of its build settings the env + fixes, and its supervisord programs beside them, run under its name: its services and its own server, which serves + its class from its venv and launches the app. Its folder under ``APPS`` is named for its id's last part.""" install, name = app.install[base.os.type], _name(app.id, "app") - layer = _Layer(f"apps/{name}", f"{APPS}/{name}", install, f"agentenv-app-{name}.conf", app=True) file, _, cls = app.module.partition(":") served = shlex.quote(f"{PurePosixPath(file).relative_to(install.files)}:{cls}") + layer = _Layer(f"apps/{name}", f"{APPS}/{name}", install, f"agentenv-app-{name}.conf", app=True, served=served, + steps=tuple((step.name, step.scope) for step in app.declaration.install_steps + if step.os is None or base.os.type in step.os)) programs = {f"app-{name}": f"venv/bin/python -m agentenv_protocol.computer.serve_app {served}", **_services(f"app-{name}", install)} _stage(context, layer, f"app {app.id}", app.files, programs, {"ENVIRONMENT_NAME": app.environment_name}) + if app.build_settings: + (context / layer.folder / BUILD_SETTINGS).parent.mkdir(exist_ok=True) + (context / layer.folder / BUILD_SETTINGS).write_text(json.dumps(app.build_settings, sort_keys=True)) return layer @@ -317,9 +341,10 @@ def _services(name: str, install: Install) -> dict[str, str]: def _stage(context: Path, layer: _Layer, what: str, files: FileArtifactUniverse, programs: dict[str, str], environment: dict[str, Any]) -> None: - """Write the ``files`` of ``what``'s folder its install names to ``layer``'s folder, a script or binary executable, - and ``programs``, commands by name, as its supervisord programs, with ``environment``.""" + """Write the ``files`` of ``what``'s folder its install names to ``layer``'s folder, a script, a binary or a file its + install runs executable, and ``programs``, commands by name, as its supervisord programs, with ``environment``.""" folder = PurePosixPath(layer.install.files) + runs = {PurePosixPath(run.script) for run in layer.install.run or []} for name, artifact in files.get_file_artifacts().items(): try: path = PurePosixPath(checked_path(name)) @@ -329,8 +354,8 @@ def _stage(context: Path, layer: _Layer, what: str, files: FileArtifactUniverse, target = context / layer.folder / path.relative_to(folder) target.parent.mkdir(parents=True, exist_ok=True) target.write_bytes(data := artifact.load()) - if data.startswith(_RUN_ONLY): - target.chmod(_EXECUTABLE) + if data.startswith(packages.RUN_ONLY) or path.relative_to(folder) in runs: + target.chmod(packages.EXECUTABLE) (context / layer.folder).mkdir(parents=True, exist_ok=True) (context / layer.conf).write_text(_programs(programs, layer.prefix, environment)) @@ -381,41 +406,69 @@ def _values(env: ComputerEnv) -> dict[str, Any]: return {**env.computer_settings, **({"resolution": f"{display['width']}x{display['height']}"} if display else {})} -def _dockerfile(source: str, layers: list[_Layer], base: ComputerArtifact, build_settings: dict, protocol: bool) -> str: - """The composed image's Dockerfile but its key's label, which the key is a digest of: each layer's files under its - prefix, its venv beside them, with this checkout's protocol when ``protocol``, and its programs; an app's server's - folder owned by the session user.""" - venvs = [layer for layer in layers if layer.install.python is not None or layer.app] - lines = [f"FROM {source}", "USER root", *(["COPY protocol /tmp/agentenv-protocol"] if protocol and venvs else [])] +def _dockerfile(source: str, layers: list[_Layer], base: ComputerArtifact, build_settings: dict, + lock: packages.Lock | None) -> str: + """The composed image's Dockerfile but its key's label, which the key is a digest of: the build-phase settings, the + lock's .debs, then each layer's files under its prefix, its venv beside them from the lock's wheels, its run scripts + and an app's install steps, each run as root or the session user, every install with the network closed; then its + programs, and an app's server's folder owned by the session user.""" + user = base.session.user + lines = [f"FROM {source}", "USER root", + *(f"ENV AGENTENV_SETTING_{name.upper()}={json.dumps(str(value))}" for name, value in build_settings.items())] + if lock is not None and lock.apt: + debs = f"/tmp/agentenv-{packages.STAGED_APT}" + lines += [f"COPY {packages.STAGED_APT}/ {debs}/", + f"RUN --network=none apt-get install -y --no-install-recommends {debs}/*.deb && rm -rf {debs}"] for layer in layers: prefix = layer.prefix lines.append(f"COPY {layer.folder}/ {prefix}/") - if layer in venvs: - lines.append(f"RUN python3 -m venv {prefix}/venv && {prefix}/venv/bin/pip install --no-cache-dir " - f"{_packages(layer, protocol)} && {prefix}/venv/bin/pip list --format=json > " - f"{prefix}/installed.json") + if _venv(layer): + staged = packages.staged_wheels(layer.folder) + wheels = f"/tmp/agentenv-{staged}" + if lock is not None and lock.wheels.get(layer.folder): + lines += [f"COPY {staged}/ {wheels}/", + f"RUN --network=none python3 -m venv {prefix}/venv && {prefix}/venv/bin/pip install " + f"--no-index --no-deps {wheels}/*.whl && {prefix}/venv/bin/pip list --format=json > " + f"{prefix}/installed.json && rm -rf {wheels}"] + else: + lines.append(f"RUN --network=none python3 -m venv {prefix}/venv && {prefix}/venv/bin/pip list " + f"--format=json > {prefix}/installed.json") if layer.app: - lines.append(f"RUN mkdir -p {prefix}/{SOCKET.parent} && chown {base.session.user} {prefix}/{SOCKET.parent}") + lines.append(f"RUN mkdir -p {prefix}/{SOCKET.parent} && chown {user} {prefix}/{SOCKET.parent}") + for run in layer.install.run or []: + lines += _as(run.scope, user, f"RUN --network=none cd {prefix} && {shlex.quote('./' + run.script)}") + for step, scope in layer.steps: + lines += _as(scope, user, f"RUN --network=none cd {prefix} && venv/bin/python -m " + f"agentenv_protocol.computer.run_install_step {layer.served} {step}") lines.append(f"COPY {layer.conf} /etc/supervisor/conf.d/{layer.conf}") checks = [f"test -d {prefix}", f"test -f /etc/supervisor/conf.d/{layer.conf}"] - if layer in venvs: + if _venv(layer): checks.append(f"test -x {prefix}/venv/bin/python") lines.append(f"RUN {' && '.join(checks)}") - if protocol and venvs: - lines.append("RUN rm -rf /tmp/agentenv-protocol") - lines += [f"ENV AGENTENV_SETTING_{name.upper()}={json.dumps(str(value))}" for name, value in build_settings.items()] - lines.append(f"USER {base.session.user}") + lines.append(f"USER {user}") return "\n".join(lines) + "\n" -def _packages(layer: _Layer, protocol: bool) -> str: - """What ``layer``'s venv installs: its requirements and, for an app, the protocol's ``app`` extra at this release, - which a driver's requirements name themselves; the protocol from this checkout's source when ``protocol``.""" - extra = "[app]" if layer.app else "" - protocols = ([f"/tmp/agentenv-protocol{extra}"] if protocol else - [f"{_PROTOCOL}{extra}=={importlib.metadata.version(_PROTOCOL)}"] if layer.app else []) - requirements = ["-r", f"{layer.prefix}/{layer.install.python}"] if layer.install.python is not None else [] - return shlex.join([*protocols, *requirements]) +def _as(scope: str, user: str, run: str) -> list[str]: + """``run`` as root, or for ``session``, as ``user``.""" + return [run] if scope == "system" else [f"USER {user}", run, "USER root"] + + +def _venv(layer: _Layer) -> bool: + """Whether ``layer`` has a venv: its requirements, or an app's own server.""" + return layer.install.python is not None or layer.app + + +def _wants(layers: list[_Layer], protocol: bool) -> list[packages.Want]: + """What each layer asks a lock for; a venv adds the protocol, for an app with its ``app`` extra: from this checkout's + source when ``protocol``, else an app's pinned to this release, which a driver's requirements name themselves.""" + wants = [] + for layer in layers: + extra = "[app]" if layer.app else "" + adds = ((f"/ctx/protocol{extra}",) if protocol else + (f"{_PROTOCOL}{extra}=={importlib.metadata.version(_PROTOCOL)}",) if layer.app else ()) + wants.append(packages.Want(layer.folder, layer.install, _venv(layer), adds if _venv(layer) else ())) + return wants def _installed(local: str, prefix: str) -> dict[str, str]: diff --git a/src/agent_env/env/envs/computer/utils/gate.py b/src/agent_env/env/envs/computer/utils/gate.py index 4aed65c2..4a8ab744 100644 --- a/src/agent_env/env/envs/computer/utils/gate.py +++ b/src/agent_env/env/envs/computer/utils/gate.py @@ -67,14 +67,16 @@ def check_app(base: ComputerArtifact, driver: ComputerDriverEnv, app: ComputerAp return problems -def check_app_settings(app: ComputerAppEnv, values: dict[str, Any], at: str) -> list[str]: - """The problems with ``values``, of ``app``'s settings by name, each named from ``at``: a setting it doesn't declare, - a secret's given as anything but a bare reference (or null, where it's optional), a required one missing, and what - the schema of its ``Settings`` refuses of the rest. A secret's reference never meets the constraints on its value, - and its value never reaches a message.""" - schema, secrets = app.declaration.settings, app.secret_settings() +def check_app_settings(app: ComputerAppEnv, values: dict[str, Any], at: str, os_type: str) -> list[str]: + """The problems with ``values``, of ``app``'s settings on ``os_type`` by name, each named from ``at``: a setting it + doesn't declare, a secret's given as anything but a bare reference (or null, where it's optional), a required one + missing, and what the schema of its ``Settings`` refuses of the rest. A secret's reference never meets the + constraints on its value, and its value never reaches a message.""" + schema, secrets = app.settings_schema(os_type), app.secret_settings(os_type) properties, required = schema.get("properties", {}), schema.get("required", []) - problems = [f"{at}.{name}: {app.id} declares no setting {name!r}" for name in values if name not in properties] + build = (app.build_settings_schema(os_type) or {}).get("properties", {}) + problems = [f"{at}.{name} is one of {app.id}'s build settings, which only another image changes" if name in build + else f"{at}.{name}: {app.id} declares no setting {name!r}" for name in values if name not in properties] problems += [f"{at}.{name} is a secret, so it takes a {SECRET_REF_PREFIX}KEY or {ENV_REF_PREFIX}NAME reference " "with no ?default, not its value" for name in values if name in secrets and not (_names_a_secret(values[name]) or values[name] is None and _optional(properties[name]))] @@ -85,6 +87,17 @@ def check_app_settings(app: ComputerAppEnv, values: dict[str, Any], at: str) -> Draft202012Validator(plain).iter_errors({k: v for k, v in values.items() if k not in secrets})] +def check_app_build_settings(app: ComputerAppEnv, values: dict[str, Any], at: str, os_type: str | None) -> list[str]: + """The problems with ``values``, of ``app``'s build settings on ``os_type`` by name, each named from ``at``: a + setting it doesn't declare as one, a required one missing, and what the schema of its ``BuildSettings`` refuses of + the rest. With no OS type, its class's.""" + schema = app.build_settings_schema(os_type) or {"type": "object", "properties": {}} + problems = [f"{at}.{name}: {app.id} declares no build setting {name!r}" for name in values + if name not in schema["properties"]] + return problems + [f"{'.'.join([at, *map(str, error.absolute_path)])}: {error.message}" + for error in Draft202012Validator(schema).iter_errors(values)] + + def _optional(setting: dict) -> bool: return any(option.get("type") == "null" for option in setting.get("anyOf", [])) diff --git a/src/agent_env/env/envs/computer/utils/install.py b/src/agent_env/env/envs/computer/utils/install.py index 6691c922..3730c8ca 100644 --- a/src/agent_env/env/envs/computer/utils/install.py +++ b/src/agent_env/env/envs/computer/utils/install.py @@ -1,17 +1,29 @@ """What a computer's layer installs on one OS, as a driver's env.toml declares it under ``[install.]`` or an app's -class as ``linux = Linux(...)``: ``files``, a folder of its own copied under the layer's prefix; ``python``, a -requirements file in that folder installed into a venv beside them; and ``services``, the programs the base's init -runs from the prefix.""" +class with ``@linux(...)``: ``files``, a folder of its own copied under the layer's prefix; ``packages``, the OS's +packages, as apt takes them; ``python``, a requirements file in that folder installed into a venv beside them; +``sources``, extra apt repositories to resolve from; ``downloads``, files fetched by URL and checked by sha256; +``run``, scripts of the folder run once those are installed; ``services``, the programs the base's init runs from the +prefix; and ``ports``, those it listens on. Everything but the files is resolved and fetched once at put, and every +build installs only what was stored then (``packages`` module). The models of packages, sources, downloads and run +scripts are the protocol's, which an app's ``@linux(...)`` uses too.""" from __future__ import annotations -from pathlib import PurePosixPath +import re +from pathlib import Path, PurePosixPath from typing import Any, Literal -from pydantic import BaseModel, ConfigDict, Field, ValidationError, field_validator +from agentenv_protocol.computer.app import AptSource, Download, LayerPath, Packages, RunScript +from pydantic import BaseModel, ConfigDict, Field, ValidationError, field_validator, model_validator from agent_env.artifact.artifacts.computer import OS_TYPES +#: The highest port a layer can listen on. +MAX_PORT = 65535 +# A requirements line that includes another requirements or constraints file, or installs one editable. +_INCLUDES = re.compile(r"-[rce]|--(requirement|constraint|editable)(=|$)") +_FIND_LINKS = re.compile(r"-f|--find-links(=|$)") + class Service(BaseModel): """A program the base's init runs: as the session user, or for ``system``, as root. A ``command`` whose program is a @@ -27,17 +39,34 @@ class Service(BaseModel): class Install(BaseModel): model_config = ConfigDict(extra="forbid") - files: str = "." - python: str | None = None + files: LayerPath = "." + packages: Packages | None = None + python: LayerPath | None = None + sources: list[AptSource] | None = None + downloads: list[Download] | None = None + run: list[RunScript] | None = None services: list[Service] = Field(min_length=1) + ports: list[int] | None = None - @field_validator("files", "python") + @field_validator("ports") @classmethod - def _relative(cls, value: str | None) -> str | None: - if value is not None and (PurePosixPath(value).is_absolute() or ".." in PurePosixPath(value).parts): - raise ValueError(f"{value!r} must be a path inside the layer's folder") + def _ports(cls, value: list[int] | None) -> list[int] | None: + if value is not None and (bad := [port for port in value if not 1 <= port <= MAX_PORT]): + raise ValueError(f"a port is 1 to {MAX_PORT}, not {bad}") + if value is not None and len(set(value)) != len(value): + raise ValueError(f"names a port twice: {value}") return value + @model_validator(mode="after") + def _sources_need_packages(self) -> Install: + if self.sources and not (self.packages and self.packages.apt): + raise ValueError("sources are apt repositories to resolve packages from, and packages names none") + return self + + def fetches(self) -> bool: + """Whether it has anything to resolve and fetch at put.""" + return bool((self.packages and self.packages.apt) or self.python is not None or self.downloads) + @field_validator("services") @classmethod def _named_once(cls, value: list[Service]) -> list[Service]: @@ -71,12 +100,48 @@ def installs(data: Any, model: type[Install] = Install) -> tuple[dict[str, Insta return found, problems -def missing(os_type: str, install: Install, files: dict) -> list[str]: - """What ``install`` names that the layer's folder, ``files`` by path, doesn't hold.""" +def folder_problems(os_type: str, install: Install, files: dict[str, Path]) -> list[str]: + """What's wrong with ``install`` against the layer's folder, ``files`` by path: what it names that the folder + doesn't hold (its folder, its requirements, its sources' keys and its scripts, each inside the folder it installs), + and each line of its requirements file that names another file: an included requirements or constraints file, an + editable install, a local path or ``file:`` URL, or a local folder of wheels. A lock's inputs are the requirements + file alone, so it would be reused when one of those changed.""" folder = PurePosixPath(install.files) problems = [] if folder != PurePosixPath(".") and not any(PurePosixPath(path).is_relative_to(folder) for path in files): problems.append(f"install.{os_type}.files: this folder has no {install.files!r} folder") - if install.python is not None and (folder / install.python).as_posix() not in files: - problems.append(f"install.{os_type}.python: {(folder / install.python).as_posix()!r} isn't in this folder") + named = [("python", install.python), *(("sources", source.key) for source in install.sources or []), + *(("run", run.script) for run in install.run or [])] + problems += [f"install.{os_type}.{field}: {(folder / path).as_posix()!r} isn't in this folder" + for field, path in named if path is not None and (folder / path).as_posix() not in files] + requirements = (folder / install.python).as_posix() if install.python is not None else None + if requirements in files: + problems += [f"install.{os_type}.python: {requirements!r} line {number}, {line.strip()!r}, names another file, " + "which the lock of a requirements file doesn't cover; list what it installs in the file itself" + for number, line in _joined(files[requirements].read_text()) if _names_a_file(line)] return problems + + +def _joined(text: str) -> list[tuple[int, str]]: + """``text``'s lines as pip reads them, a line ending in a backslash joined to the next, each with the number of the + line it starts on.""" + lines, start, parts = [], None, [] + for number, line in enumerate(text.splitlines(), 1): + start = start or number + if line.endswith("\\") and not line.lstrip().startswith("#"): + parts.append(line[:-1]) + continue + lines.append((start, "".join(parts) + line)) + start, parts = None, [] + return lines + ([(start, "".join(parts))] if parts else []) + + +def _names_a_file(line: str) -> bool: + words = re.sub(r"(^|\s)#.*", "", line).split() + if not words: + return False + if _INCLUDES.match(words[0]): + return True + if _FIND_LINKS.match(words[0]): + return "://" not in line + return words[0].startswith((".", "/", "~", "file:")) or re.search(r"@\s*file:", line) is not None diff --git a/src/agent_env/env/envs/computer/utils/packages.py b/src/agent_env/env/envs/computer/utils/packages.py new file mode 100644 index 00000000..b3016b24 --- /dev/null +++ b/src/agent_env/env/envs/computer/utils/packages.py @@ -0,0 +1,287 @@ +"""Fetch at put, install from the store: what a computer env's layers ask for beyond their own files (apt packages, +Python requirements and downloads) is resolved once for each arch in a container of the base (``resolve_in_base``), +every file is stored in the object store by its sha256, and a lock lists them (``Lock``). The lock is stored by a +digest of what it was resolved from (each layer's packages, sources, requirements file and downloads, the base, the +arch, the request the resolver is given, the resolver itself and the lock's own shape), so any env with the same inputs +reuses it. A build stages only the lock's files and installs them with the network closed (``composer``).""" + +from __future__ import annotations + +import hashlib +import json +import os +import shutil +import subprocess +from dataclasses import dataclass +from pathlib import Path + +import httpx +from pydantic import BaseModel + +from agent_env.config import get_config +from agent_env.env.envs.computer.utils.install import Install +from agent_env.store.base import ObjectAlreadyExistsError + +#: Where locks and their files are kept, under the configured key prefix, in the store the env's id routes to. +PREFIX = "agentenv-packages" +RESOLVER = Path(__file__).with_name("resolve_in_base.py") +#: Where a build's context holds a lock's .debs; each venv's wheels are at ``staged_wheels``. +STAGED_APT = "packages/apt" +_RESOLVE_TIMEOUT_S = 3600 +_DOWNLOAD_TIMEOUT_S = 600 +_CHUNK = 1 << 20 +#: What a file starts with when it's only ever run: a script or a Linux binary, made executable where it's staged, +#: since the stores keep no file modes. +RUN_ONLY = (b"#!", b"\x7fELF") +EXECUTABLE = 0o755 + + +class ResolveError(RuntimeError): + """Resolving or fetching failed; ``log`` holds what the resolver printed.""" + + def __init__(self, message: str, log: str = ""): + super().__init__(message) + self.log = log + + +class LockedFile(BaseModel): + file: str + sha256: str + size: int + + +class LockedDeb(LockedFile): + package: str + version: str + arch: str + + +class LockedDownload(LockedFile): + url: str + to: str + + +class Lock(BaseModel): + inputs: str + arch: str + apt: list[LockedDeb] + wheels: dict[str, list[LockedFile]] + downloads: dict[str, list[LockedDownload]] + + def digest(self) -> str: + """The part of a build key that says exactly what the lock installs.""" + return "sha256:" + hashlib.sha256(self.model_dump_json().encode()).hexdigest() + + +@dataclass(frozen=True) +class Want: + """A layer's part of a lock: its folder in the build context, its install, and when it has a venv, the requirements + the venv adds to its own (the protocol).""" + + folder: str + install: Install + venv: bool + adds: tuple[str, ...] = () + + @property + def name(self) -> str: + return _slug(self.folder) + + +def inputs_digest(wants: list[Want], context: Path, base: str, arch: str) -> str | None: + """The digest of everything a lock of ``wants`` for ``arch`` is resolved from, their folders staged in ``context`` + and ``base`` the base's identity; None when they ask for nothing.""" + if not any(want.install.fetches() or want.venv for want in wants): + return None + layers = [] + for want in wants: + install, folder = want.install, context / want.folder + layers.append({ + "folder": want.folder, "apt": install.packages.apt if install.packages else [], + "sources": [{**source.model_dump(), "key": _sha256(folder / source.key)} for source in install.sources or []], + "python": _sha256(folder / install.python) if install.python is not None else None, + "venv": want.venv, "adds": list(want.adds), + "downloads": [download.model_dump() for download in _downloads(install, arch)]}) + protocol = context / "protocol" + return "sha256:" + hashlib.sha256(json.dumps({ + "layers": layers, "base": base, "arch": arch, "request": _request(wants, arch), "resolver": _sha256(RESOLVER), + "lock": _lock_shape(), "protocol": _folder_digest(protocol) if protocol.is_dir() else None, + }, sort_keys=True).encode()).hexdigest() + + +def stored(inputs: str, store_id: str) -> Lock | None: + """The lock resolved from ``inputs``, when one is stored.""" + store, prefix = _store(store_id) + key = _lock_key(prefix, inputs) + return Lock.model_validate_json(store.read(key)) if store.exists(key) else None + + +def resolve(inputs: str, wants: list[Want], context: Path, source: str, arch: str, store_id: str, + work: Path) -> tuple[Lock, dict[str, Path]]: + """Resolve ``wants`` for ``arch`` in a container of ``source``, the base's image, with their folders and the + protocol's source from ``context``, into ``work``; fetch their downloads; store every file and the lock. The lock, + and where each file this resolve fetched is under ``work``, by its sha256, for a build that follows to stage it from + there rather than the store.""" + (store, prefix), fetched = _store(store_id), {} + result, out = _run_resolver(_request(wants, arch), context, source, arch, work) + apt = [LockedDeb(**{**deb, **_put(store, prefix, fetched, _written(out, "apt", deb["file"]))}) + for deb in result["apt"]] + wheels = {want.folder: [LockedFile(**_put(store, prefix, fetched, _written(out, "wheels", want.name, wheel))) + for wheel in result["wheels"][want.name]] for want in wants if want.venv} + downloads = {want.folder: [LockedDownload(url=download.url, to=download.to, **_put( + store, prefix, fetched, _fetch(download.url, download.sha256, work))) + for download in _downloads(want.install, arch)] + for want in wants if _downloads(want.install, arch)} + lock = Lock(inputs=inputs, arch=arch, apt=apt, wheels=wheels, downloads=downloads) + key = _lock_key(prefix, inputs) + try: + store.put(key, lock.model_dump_json().encode(), content_type="application/json") + except ObjectAlreadyExistsError: + return Lock.model_validate_json(store.read(key)), fetched + return lock, fetched + + +def stage(lock: Lock, context: Path, store_id: str, fetched: dict[str, Path]) -> None: + """Write the lock's files into ``context`` for a build: .debs under ``STAGED_APT``, each venv's wheels under + ``staged_wheels``, and each layer's downloads at their places in its folder, a script or binary executable; each + linked from ``fetched``, the files a resolve just fetched by sha256, or else downloaded from the store and refused + unless its sha256 is the one the lock names.""" + store, prefix = _store(store_id) + for deb in lock.apt: + _get(store, prefix, fetched, deb.sha256, context / STAGED_APT / deb.file) + for folder, wheels in lock.wheels.items(): + for wheel in wheels: + _get(store, prefix, fetched, wheel.sha256, context / staged_wheels(folder) / wheel.file) + for folder, downloads in lock.downloads.items(): + for download in downloads: + target = context / folder / download.to + _get(store, prefix, fetched, download.sha256, target) + with target.open("rb") as file: + if file.read(4).startswith(RUN_ONLY): + target.chmod(EXECUTABLE) + + +def staged_wheels(folder: str) -> str: + """Where a build's context holds the wheels of the venv of the layer in ``folder``.""" + return f"packages/wheels/{_slug(folder)}" + + +def _request(wants: list[Want], arch: str) -> dict: + """What the resolver in the base resolves, by its paths there: the build context at ``/ctx``.""" + apt = list(dict.fromkeys(name for want in wants for name in (want.install.packages.apt if want.install.packages + else []))) + sources = [{**source.model_dump(), "key": f"/ctx/{want.folder}/{source.key}"} + for want in wants for source in want.install.sources or []] + python = [{"name": want.name, "folder": f"/ctx/{want.folder}", "requirements": want.install.python, + "add": list(want.adds)} for want in wants if want.venv] + return {"apt": apt, "sources": sources, "python": python} + + +def _run_resolver(request: dict, context: Path, source: str, arch: str, work: Path) -> tuple[dict, Path]: + """Run the resolver in a container of ``source`` for ``arch``, as root, with the build context read-only at /ctx; + what it wrote, and the folder it wrote it in.""" + out = work / "out" + out.mkdir(parents=True) + (out / "request.json").write_text(json.dumps({**request, "owner": [os.getuid(), os.getgid()]})) + shutil.copy(RESOLVER, out / "resolve.py") + try: + ran = subprocess.run(["docker", "run", "--rm", "--platform", f"linux/{arch}", "--user", "root", + "--entrypoint", "python3", "-v", f"{context}:/ctx:ro", "-v", f"{out}:/out", source, + "/out/resolve.py", "/out/request.json", "/out"], + capture_output=True, text=True, timeout=_RESOLVE_TIMEOUT_S) + except subprocess.TimeoutExpired as e: + raise ResolveError(f"resolving took over {_RESOLVE_TIMEOUT_S} s", str(e.stdout or "")) from None + if ran.returncode != 0: + raise ResolveError("resolving the packages failed", ran.stdout + ran.stderr) + return json.loads((out / "result.json").read_text()), out + + +def _fetch(url: str, sha256: str, work: Path) -> Path: + """Download ``url`` into ``work``, refused unless its sha256 is ``sha256``.""" + target, hashed = work / "downloads" / sha256, hashlib.sha256() + target.parent.mkdir(parents=True, exist_ok=True) + try: + with httpx.stream("GET", url, follow_redirects=True, timeout=_DOWNLOAD_TIMEOUT_S) as response: + response.raise_for_status() + with target.open("wb") as file: + for chunk in response.iter_bytes(_CHUNK): + hashed.update(chunk) + file.write(chunk) + except httpx.HTTPError as e: + raise ResolveError(f"downloading {url} failed: {e}") from None + if hashed.hexdigest() != sha256: + raise ResolveError(f"{url} has the sha256 {hashed.hexdigest()}, not the {sha256} its download names") + return target + + +def _downloads(install: Install, arch: str) -> list: + return [download for download in install.downloads or [] if download.arch in (None, arch)] + + +def _written(out: Path, *parts: str) -> Path: + """The file the resolver says it wrote at ``parts`` under ``out``, refused unless it's a plain file there: a + dependency's build code, run in the container, could leave a link to one of this machine's files instead.""" + path = out.joinpath(*parts) + if path.is_symlink() or not path.is_file() or not path.resolve().is_relative_to(out.resolve()): + raise ResolveError(f"the resolver's {'/'.join(parts)!r} isn't a plain file it wrote") + return path + + +def _put(store, prefix: str, fetched: dict[str, Path], path: Path) -> dict: + """Store the file at ``path`` by its sha256 unless it's there, and add it to ``fetched``; its name, sha256 and + size.""" + sha256 = _sha256(path) + key = f"{prefix}/files/{sha256}" + if not store.exists(key): + try: + store.put_file(key, str(path)) + except ObjectAlreadyExistsError: + pass + fetched[sha256] = path + return {"file": path.name, "sha256": sha256, "size": path.stat().st_size} + + +def _get(store, prefix: str, fetched: dict[str, Path], sha256: str, target: Path) -> None: + target.parent.mkdir(parents=True, exist_ok=True) + if sha256 in fetched: + target.unlink(missing_ok=True) + os.link(fetched[sha256], target) + return + key = f"{prefix}/files/{sha256}" + store.download_to_file(store.object_url(key), str(target)) + if (found := _sha256(target)) != sha256: + raise RuntimeError(f"the store's {key} isn't the file the lock names: its sha256 is {found}") + + +def _store(store_id: str): + config = get_config() + return config.get_object_store_for(store_id), f"{config.get_artifact_key_prefix()}{PREFIX}" + + +def _sha256(path: Path) -> str: + hashed = hashlib.sha256() + with path.open("rb") as file: + while chunk := file.read(_CHUNK): + hashed.update(chunk) + return hashed.hexdigest() + + +def _folder_digest(folder: Path) -> str: + hashed = hashlib.sha256() + for path in sorted(p for p in folder.rglob("*") if p.is_file()): + hashed.update(path.relative_to(folder).as_posix().encode() + b"\0" + bytes.fromhex(_sha256(path))) + return hashed.hexdigest() + + +def _lock_shape() -> dict[str, dict[str, str]]: + """The fields of a lock and their types, one of its inputs, so a lock stored in another shape is resolved again.""" + return {model.__name__: {name: str(field.annotation) for name, field in model.model_fields.items()} + for model in (Lock, LockedFile, LockedDeb, LockedDownload)} + + +def _lock_key(prefix: str, inputs: str) -> str: + return f"{prefix}/locks/{inputs.removeprefix('sha256:')}.json" + + +def _slug(folder: str) -> str: + return folder.replace("/", "-") diff --git a/src/agent_env/env/envs/computer/utils/params.py b/src/agent_env/env/envs/computer/utils/params.py index 4edc8067..3e765cf5 100644 --- a/src/agent_env/env/envs/computer/utils/params.py +++ b/src/agent_env/env/envs/computer/utils/params.py @@ -53,7 +53,8 @@ def checked(env: ComputerEnv, env_params: Optional[dict]) -> tuple[ComputerParam problems += [f"env_params.app_settings.{name}: {env.id} has no app {name!r}" + (f"; its apps are {', '.join(apps)}" if apps else "") for name in params.app_settings if name not in apps] for name, app in apps.items(): - problems += gate.check_app_settings(app, params.app_settings.get(name, {}), f"env_params.app_settings.{name}") + problems += gate.check_app_settings(app, params.app_settings.get(name, {}), f"env_params.app_settings.{name}", + env.computer.os.type) return params, values, problems diff --git a/src/agent_env/env/envs/computer/utils/resolve_in_base.py b/src/agent_env/env/envs/computer/utils/resolve_in_base.py new file mode 100644 index 00000000..6425a495 --- /dev/null +++ b/src/agent_env/env/envs/computer/utils/resolve_in_base.py @@ -0,0 +1,73 @@ +"""Resolve what a computer env's layers ask for, run by the composer at put inside a container of the computer's base +for one arch (``packages`` module), so it resolves against the base's own installed packages: apt packages and their +dependencies, from the OS's repositories and the layers' sources, downloaded into ``/apt``; each layer's Python +requirements as wheels into ``/wheels/``. Standard library only, since it runs on the base's Python. +Usage: ``python3 resolve_in_base.py ``; it writes ``/result.json``.""" + +import json +import os +import shutil +import subprocess +import sys +from pathlib import Path + +KEYRINGS = Path("/etc/apt/keyrings") +SOURCES = Path("/etc/apt/sources.list.d") +ARMORED = b"-----BEGIN PGP PUBLIC KEY BLOCK-----" + + +def main(request_path: str, out: str) -> None: + request, folder = json.loads(Path(request_path).read_text()), Path(out) + result = {"apt": _apt(request["apt"], request["sources"], folder / "apt") if request["apt"] else [], + "wheels": {layer["name"]: _wheels(layer, folder / "wheels" / layer["name"]) for layer in request["python"]}} + (folder / "result.json").write_text(json.dumps(result)) + uid, gid = request["owner"] + for path in [folder, *folder.rglob("*")]: + os.chown(path, uid, gid) + + +def _apt(packages: list[str], sources: list[dict], dest: Path) -> list[dict]: + """Download ``packages`` with every dependency the base doesn't have, no recommends, from the OS's repositories and + ``sources``, each signed by its key, and install them here too, so the wheels built next have the build tools and + headers they name; each .deb with its package, version and arch.""" + KEYRINGS.mkdir(parents=True, exist_ok=True) + for i, source in enumerate(sources): + key = Path(source["key"]).read_bytes() + ring = KEYRINGS / f"agentenv-{i}{'.asc' if key.lstrip().startswith(ARMORED) else '.gpg'}" + ring.write_bytes(key) + components = f"Components: {' '.join(source['components'])}\n" if source["components"] else "" + (SOURCES / f"agentenv-{i}.sources").write_text( + f"Types: deb\nURIs: {source['apt']}\nSuites: {source['suite']}\n{components}Signed-By: {ring}\n") + env = {**os.environ, "DEBIAN_FRONTEND": "noninteractive"} + (dest / "partial").mkdir(parents=True) + subprocess.run(["apt-get", "update", "-q"], check=True, env=env) + archives = ["-o", f"Dir::Cache::Archives={dest}/"] + subprocess.run(["apt-get", "install", "-y", "-q", "--download-only", "--no-install-recommends", *archives, + *packages], check=True, env=env) + subprocess.run(["apt-get", "install", "-y", "-q", "--no-install-recommends", *archives, *packages], check=True, + env=env) + shutil.rmtree(dest / "partial") + (dest / "lock").unlink(missing_ok=True) + debs = [] + for deb in sorted(dest.glob("*.deb")): + fields = subprocess.run(["dpkg-deb", "-f", str(deb), "Package", "Version", "Architecture"], check=True, + capture_output=True, text=True).stdout + found = dict(line.split(": ", 1) for line in fields.splitlines()) + debs.append({"file": deb.name, "package": found["Package"], "version": found["Version"], + "arch": found["Architecture"]}) + return debs + + +def _wheels(layer: dict, dest: Path) -> list[str]: + """The layer's requirements, and what its venv adds, as wheels for this Python and arch: a package published only + as source is built into one here, once.""" + venv = Path("/tmp/agentenv-resolve") / layer["name"] + subprocess.run([sys.executable, "-m", "venv", str(venv)], check=True) + requirements = ["-r", layer["requirements"]] if layer["requirements"] else [] + subprocess.run([str(venv / "bin" / "pip"), "wheel", "--no-cache-dir", "--wheel-dir", str(dest), *requirements, + *layer["add"]], check=True, cwd=layer["folder"]) + return sorted(path.name for path in dest.glob("*.whl")) + + +if __name__ == "__main__": + main(*sys.argv[1:]) diff --git a/src/agent_env/providers/env_providers/env_computer_provider.py b/src/agent_env/providers/env_providers/env_computer_provider.py index 5dc11e48..e4903e74 100644 --- a/src/agent_env/providers/env_providers/env_computer_provider.py +++ b/src/agent_env/providers/env_providers/env_computer_provider.py @@ -96,7 +96,8 @@ async def _boot(self, env: ComputerEnv, builds: list[tuple[str, str, dict]], par booted = await boot( sandbox_provider, image=build["image"], port=env.driver.port, network_policy=policy, environment=_environment(env, computer, values, params), timeout=ttl_seconds, attribution=attribution, - disk_size_gb=disk_size_gb, cpu=cpu, memory_mb=memory_mb, ready_s=ready_timeout(env.composed_apps())) + disk_size_gb=disk_size_gb, cpu=cpu, memory_mb=memory_mb, + ready_s=ready_timeout(env.composed_apps(), env.computer.os.type)) self._container_sandboxes.append(booted.sandbox) self._environment_sandboxes[name] = booted.sandbox try: @@ -136,7 +137,8 @@ async def reboot(self, env: ComputerEnv, deployed: DeployedSandboxEnv) -> None: env=_environment(env, computer, values, params)) except NotImplementedError as e: # a reset the env supports, which this provider can't carry out raise RuntimeError(f"'{env.id}' can't be reset on {type(sandbox_provider).__name__}: {e}") from None - _require_name(env, await ready(deployed.environment_url, ready_timeout(env.composed_apps()))) + card = await ready(deployed.environment_url, ready_timeout(env.composed_apps(), env.computer.os.type)) + _require_name(env, card) await _require_display(deployed, values.get("resolution")) @@ -161,7 +163,7 @@ def _environment(env: ComputerEnv, computer: ComputerArtifact, values: dict, params: computer_params.ComputerParams) -> dict[str, str]: return {**settings_environment(computer, values, environment_name=env.environment_name, driver_settings=params.driver_settings), - **app_settings_environment(env.composed_apps(), params.app_settings)} + **app_settings_environment(env.composed_apps(), params.app_settings, env.computer.os.type)} def _require_name(env: ComputerEnv, card: dict) -> None: diff --git a/tst/integration/computer/app/artifacts/check_notes/run.py b/tst/integration/computer/app/artifacts/check_notes/run.py index 533c5a6d..e91675f1 100644 --- a/tst/integration/computer/app/artifacts/check_notes/run.py +++ b/tst/integration/computer/app/artifacts/check_notes/run.py @@ -1,11 +1,14 @@ """What the notes app's service wrote, which it can only once its settings reached it, the name the computer's card gives and what its readiness says of the app, asked on loopback from inside the computer, the launcher its server added -to the session user's menu and the apps launched since the computer booted.""" +to the session user's menu, the apps launched since the computer booted, and what its install left: its run script's +output of the apt packages it installed, its PPA's among them, its download, and its install step's file.""" import json import time import urllib.request from pathlib import Path +PREFIX = Path("/opt/agentenv/apps/notes") + def run(input): written = Path.home() / ".notes.json" @@ -20,4 +23,8 @@ def run(input): return {**json.loads(written.read_text()), "card": card["name"], "app_ready": [check for check in ready["checks"] if check["name"].startswith("app ")], "launchers": sorted(p.name for p in (Path.home() / ".local/share/applications").glob("agentenv-app-*")), - "launched": sorted(p.name for p in Path("/run/agentenv/apps").iterdir())} + "launched": sorted(p.name for p in Path("/run/agentenv/apps").iterdir()), + "hello": (PREFIX / "hello.txt").read_text().strip(), + "fastfetch": (PREFIX / "fastfetch.txt").read_text().splitlines(), + "license": (PREFIX / "LICENSE.txt").read_text().startswith("A. HISTORY OF THE SOFTWARE"), + "installed": (Path.home() / ".notes-installed").read_text()} diff --git a/tst/integration/computer/app/envs/notes/app.py b/tst/integration/computer/app/envs/notes/app.py index 8e5f75b0..11933a1b 100644 --- a/tst/integration/computer/app/envs/notes/app.py +++ b/tst/integration/computer/app/envs/notes/app.py @@ -1,24 +1,48 @@ """A notes app for the computer app tests: its service writes the settings a deploy gave it to ~/.notes.json, and its -own server launches its window once the desktop session is up; it's ready once both have happened. It requires nothing -of its own, so its service and window run in the venv every app gets for its server.""" +own server launches its window once the desktop session is up; it's ready once both have happened. It requires no +Python of its own, so its service and window run in the venv every app gets for its server. Its install takes an apt +package from Ubuntu's archive and one only a Launchpad PPA has, which its run script uses, a download checked by sha256, +and an install step run as the session user on the build setting its env.toml fixes. On Linux its settings gain a font, +and its install step and ready check are Linux's.""" from pathlib import Path from pydantic import BaseModel, SecretStr -from agentenv_protocol.computer import ComputerAppEnvironment, Launch, Linux, Ready, Service, computer_app, ready_check +from agentenv_protocol.computer import (AptSource, ComputerAppEnvironment, Download, Launch, Packages, Ready, RunScript, + Service, computer_app, install_step, linux, ready_check) + +LICENSE = "https://raw.githubusercontent.com/python/cpython/v3.12.0/LICENSE" +FASTFETCH = "https://ppa.launchpadcontent.net/zhangsongcui3371/fastfetch/ubuntu" + + +class LinuxSettings(BaseModel): + font: str = "DejaVu Sans" @computer_app(supports=[{"os": {"type": "linux", "display": ["x11"]}, "arch": ["amd64", "arm64"]}]) +@linux(packages=Packages(apt=["hello", "fastfetch"]), + sources=[AptSource(apt=FASTFETCH, suite="noble", components=["main"], key="fastfetch.asc")], + downloads=[Download(url=LICENSE, to="LICENSE.txt", + sha256="3b2f81fe21d181c499c59a256c8e1968455d6689d269aa85373bfb6af41da3bf")], + run=[RunScript(script="setup.sh")], + services=[Service(name="notes", command="venv/bin/python serve.py")], + launch=Launch(at="boot", command=["venv/bin/python", "window.py"], window={"class": "Notes"}, + ready=Ready(window=True)), + settings=LinuxSettings) class Notes(ComputerAppEnvironment): - linux = Linux(services=[Service(name="notes", command="venv/bin/python serve.py")]) - launch = Launch(at="boot", command=["venv/bin/python", "window.py"], window={"class": "Notes"}, - ready=Ready(window=True)) skill = "skill.md" class Settings(BaseModel): signature: str = "-- notes" token: SecretStr = SecretStr("") - @ready_check + class BuildSettings(BaseModel): + edition: str = "home" + + @install_step(scope="session", os="linux") + def remember(self) -> None: + (Path.home() / ".notes-installed").write_text(self.build_settings.edition) + + @ready_check(os="linux") def written(self) -> bool: return (Path.home() / ".notes.json").exists() diff --git a/tst/integration/computer/app/envs/notes/env.toml b/tst/integration/computer/app/envs/notes/env.toml index 61191b1c..eb40ca4d 100644 --- a/tst/integration/computer/app/envs/notes/env.toml +++ b/tst/integration/computer/app/envs/notes/env.toml @@ -2,3 +2,4 @@ type = "computer_app" module = "app.py:Notes" computer = "ubuntu-24.04" driver = "reference" +build_settings = { edition = "pro" } diff --git a/tst/integration/computer/app/envs/notes/fastfetch.asc b/tst/integration/computer/app/envs/notes/fastfetch.asc new file mode 100644 index 00000000..c54a3728 --- /dev/null +++ b/tst/integration/computer/app/envs/notes/fastfetch.asc @@ -0,0 +1,30 @@ +-----BEGIN PGP PUBLIC KEY BLOCK----- +Comment: Hostname: +Version: Hockeypuck 2.2 + +xsFNBGXC33sBEADqChRifM9aJb4sJnn9jKV4mdt9Rj90OAYfySPuVyVX+dDKWP8X +ATjtDDNJeROme4+05iSt1SAey9W0ugYo8ft2LeMCjWlog6mlM3EPJ4UelEVBOGpF +su/cgTZbu0A6bHIliFeQhDavd897Dr8ZH9+QIPumNyHB+mtd+U/S9qggZtWs6seu +WJo2eRYXC4ubJ52rjpCAT2bSM2Rn7YATAKxYl35+ef4ytb5i+wkgUVD55AuYzNL7 +eMzSOY4iYF9SWUrYxtNKq2AH6IZ6G49Vjt0yDcH8wl2wImvKcI4hwQYbxHXwBj2l +E7msT65gHyFocRrTp1XvrOGms6iqE3GjXy7CwiWKtxK/DLFx8Pf2A27E+95OKsrj +Cs9g+g5H5625QSp7xgocaNtmHz+gTEGkstDIiAZQ2/tp0jN4sPpFKEp03oMxr9+8 +iON68SD0lPRRfPBwQat0CkkUK1YInnw1ZhPQJJk8XW8mxPYsbQ+hNofUo0gN1dX3 +T/1RNi8x2zDiNP5lyoSzX0V2G7kIvL+G0xUnhMauNXqQcLXfKukDL9Vf4q/TWmAS +wQhJ2oCeRrdlOWhBHOoFNUSpt2PxRHqBd5SMCh29HIdN2CLT3XitPj10ozyrXVGZ +FLKL1uZqZar03ofmw/qNYXG0smG12LsjCjWDtX94Cg+/Fl1dVYfC3ziPzQARAQAB +zRtMYXVuY2hwYWQgUFBBIGZvciBDYXJ0ZXIgTGnCwY4EEwEKADgWIQTrZe4Z2ALz +6xoTz+R+Lly01IZfIQUCZcLfewIbAwULCQgHAgYVCgkICwIEFgIDAQIeAQIXgAAK +CRB+Lly01IZfIbRvD/0Zx9paqlDJWwCTd+1b9kZPOWeylBvQ5A8rvJb7RyccX6AR +gXj1zWkc76OV1A3TiwVqsFhX357jG7TJ2HkAcld7EwhbKjne3nmtCLUwSFdJERP2 +dpobBAtmZV7g4dEHlHH3nFRcowrsAFwlHTp72Iak39WVjxR/XQDX9yeTg8H0hJZ9 +veh8slwcPw0ORevqXnOrKYM3uolLoBEUXe2Y4Zc/gBqN6OnK6uv0ISfG/zG3yOJt +Z1HJ6ITamqHv8QDjg2rbxYt2VtvuOe/8T34nI6otdtJUDNDllLNn/5Td1YQsdgUg +Rgf9bMM7X8LTJW/2zc++I1XCJToVFuXdD/IcIB+e10Y8yrhBjer3R4wtoW7ubXG9 +SmURCLfu9JxmDjl5T+Fsg4YDlSIih02P3SitwPWG0AvqYWDsKGfx5tF5/28wjHoM +bsi1xxAxnlYuzcZT/Xb6ZO4wMcV5QiRbfVHAACt69vyABXfFlqJ8ScSt8pGsR9bl +dCRlrLFQh13NDnSJQtnO3mx4eF1ir1iO4hjJFTDhhE8QbImS3w1hEhZuHGVXFo8a +xsChEDlop6GNFFUUnOwS3X+sKVQjL9Emiv1DkXTSnJmDwKQyOr6T3uTgzT5o9Gt8 +SrnkCBlxrqZHzaUJuSGV0YxxWUAexlj4nlYwoZz5uXTNikhD38q+O9ZNG4SQTg== +=yPNc +-----END PGP PUBLIC KEY BLOCK----- diff --git a/tst/integration/computer/app/envs/notes/serve.py b/tst/integration/computer/app/envs/notes/serve.py index 44620ad7..536de5db 100644 --- a/tst/integration/computer/app/envs/notes/serve.py +++ b/tst/integration/computer/app/envs/notes/serve.py @@ -6,6 +6,7 @@ notes = Notes() Path.home().joinpath(".notes.json").write_text(json.dumps( - {"signature": notes.settings.signature, "token": notes.settings.token.get_secret_value()})) + {"signature": notes.settings.signature, "token": notes.settings.token.get_secret_value(), + "edition": notes.build_settings.edition, "font": notes.settings.font})) while True: time.sleep(3600) diff --git a/tst/integration/computer/app/envs/notes/setup.sh b/tst/integration/computer/app/envs/notes/setup.sh new file mode 100644 index 00000000..075b5088 --- /dev/null +++ b/tst/integration/computer/app/envs/notes/setup.sh @@ -0,0 +1,6 @@ +#!/bin/sh +# Run as root once the app's packages are in: hello is one of them, and fastfetch the one only its PPA has. +set -e +hello > hello.txt +dpkg-query -W -f '${Version}\n' fastfetch > fastfetch.txt +fastfetch --version >> fastfetch.txt diff --git a/tst/integration/computer/computer_app_test.py b/tst/integration/computer/computer_app_test.py index 3d1a1dfd..fb4c09da 100644 --- a/tst/integration/computer/computer_app_test.py +++ b/tst/integration/computer/computer_app_test.py @@ -1,8 +1,9 @@ """A computer app deployed alone through ``agent-env run``: a one-app computer of the computer and driver its env.toml names, its card named after it, its service running from its folder in the venv its own server runs in, on the settings the deploy gave it, a secret resolved at boot, its window launched once by its server as the desktop session -comes up, and the computer ready only once its server says its window and its @ready_check are. Needs Docker and the -local registry on localhost:5000.""" +comes up, and the computer ready only once its server says its window and its @ready_check are; its apt packages, +one from a PPA, its download, run script and install step on its build setting, fetched at put, installed offline. +Needs Docker and the local registry on localhost:5000.""" from pathlib import Path @@ -26,7 +27,11 @@ def test_an_app_deployed_alone_runs_its_service_on_its_settings_on_a_computer_na context = run_context(ran.output) (deployed,) = context["deployed_envs"] assert deployed["env_id"].endswith("/notes") and deployed["environment_card"]["name"] == "notes" - assert context["metadata"]["script_results"]["service"] == { - "signature": "-- Ada", "token": "t0ken", "card": "notes", + service = context["metadata"]["script_results"]["service"] + version, fastfetch = service.pop("fastfetch") + assert version.endswith("~noble") and fastfetch.startswith("fastfetch ") + assert service == { + "signature": "-- Ada", "token": "t0ken", "edition": "pro", "font": "DejaVu Sans", "card": "notes", "app_ready": [{"name": "app notes", "ok": True, "detail": "ready"}], - "launchers": ["agentenv-app-notes.desktop"], "launched": ["notes"]} + "launchers": ["agentenv-app-notes.desktop"], "launched": ["notes"], "hello": "Hello, world!", "license": True, + "installed": "pro"} diff --git a/tst/unit/bundle/computer_app_env_test.py b/tst/unit/bundle/computer_app_env_test.py index 2f9fe7b5..8457d5c2 100644 --- a/tst/unit/bundle/computer_app_env_test.py +++ b/tst/unit/bundle/computer_app_env_test.py @@ -5,7 +5,7 @@ import json import pytest -from agentenv_protocol.computer import Launch, Linux, Service +from agentenv_protocol.computer import Launch, Service from agentenv_protocol.computer.app import AppSkill, InstallStep from agent_env.bundle import BundleError @@ -22,19 +22,22 @@ from pydantic import BaseModel, SecretStr from agentenv_protocol import add_data, get_data -from agentenv_protocol.computer import ComputerAppEnvironment, Launch, Linux, Service, computer_app, install_step +from agentenv_protocol.computer import ComputerAppEnvironment, Launch, Service, computer_app, install_step, linux @computer_app(supports={supports}, permissions=["camera"]) +@linux(python="requirements.txt", services=[Service(name="imap", command="venv/bin/python imap.py")], + launch=Launch(command=["thunderbird"], window={{"class": "thunderbird"}}), skill={linux_skill!r}) class Mail(ComputerAppEnvironment): - linux = Linux(python="requirements.txt", services=[Service(name="imap", command="venv/bin/python imap.py")]) - launch = Launch(command=["thunderbird"], window={{"class": "thunderbird"}}) skill = {skill!r} class Settings(BaseModel): signature: str = "" password: SecretStr = SecretStr("") + class BuildSettings(BaseModel): + edition: str = "home" + @install_step(scope="session") def profile(self): not_installed_here.profile() @@ -53,9 +56,11 @@ def bundle_dir(tmp_path, monkeypatch, local_stores, cli_routing): return tmp_path / "apps" -def _bundle(root, toml=TOML, supports=LINUX, skill="skill.md", **extra): - files = {"envs/mail/env.toml": toml, "envs/mail/app.py": APP.format(supports=supports, skill=skill), - "envs/mail/requirements.txt": "agentenv-framework-protocol\n", "envs/mail/skill.md": "# Mail\n"} +def _bundle(root, toml=TOML, supports=LINUX, skill="skill.md", linux_skill="skill-linux.md", **extra): + files = {"envs/mail/env.toml": toml, + "envs/mail/app.py": APP.format(supports=supports, skill=skill, linux_skill=linux_skill), + "envs/mail/requirements.txt": "agentenv-framework-protocol\n", "envs/mail/skill.md": "# Mail\n", + "envs/mail/skill-linux.md": "# Mail on Linux\n"} return layout(root, {**files, **extra, "tasks/t.json": json.dumps([{"id": "agent", "type": "deploy_agent", "env_ids": ["mail"]}])}) @@ -68,19 +73,23 @@ def _problems(root): @pytest.mark.asyncio async def test_an_app_folder_is_written_with_its_files_and_what_its_class_declares(bundle_dir): - _bundle(bundle_dir) + _bundle(bundle_dir, toml=f'{TOML}build_settings = {{ edition = "pro" }}\n') materialize(plan_of(bundle_dir)) env = Env.get(f"{ROOT}/mail") assert isinstance(env, ComputerAppEnv) and (env.environment_name, env.module) == ("mail", "app.py:Mail") - assert sorted(env.files.file_artifact_ids) == ["app.py", "requirements.txt", "skill.md"] + assert sorted(env.files.file_artifact_ids) == ["app.py", "requirements.txt", "skill-linux.md", "skill.md"] declared = env.declaration assert (declared.supports, declared.permissions) == (LINUX, ["camera"]) - assert declared.install == {"linux": Linux(python="requirements.txt", - services=[Service(name="imap", command="venv/bin/python imap.py")])} - assert declared.launch == Launch(command=["thunderbird"], window={"class": "thunderbird"}) + assert declared.os["linux"].install == {"python": "requirements.txt", "files": ".", "services": [ + Service(name="imap", command="venv/bin/python imap.py").model_dump()]} + assert [(service.name, service.command) for service in env.install["linux"].services] == [ + ("imap", "venv/bin/python imap.py")] + assert (declared.os["linux"].launch, declared.os["linux"].skill) == ( + Launch(command=["thunderbird"], window={"class": "thunderbird"}), AppSkill(file="skill-linux.md")) assert declared.settings["properties"]["password"]["writeOnly"] is True + assert list(declared.build_settings["properties"]) == ["edition"] and env.build_settings == {"edition": "pro"} assert (declared.skill, declared.data) == (AppSkill(file="skill.md"), ["data/add", "data/get"]) assert declared.install_steps == [InstallStep(name="profile", scope="session")] assert (env.computer, env.driver) == (None, None) @@ -99,8 +108,11 @@ async def test_an_app_folder_is_written_with_its_files_and_what_its_class_declar (TOML, {"envs/mail/requirements.txt": None}, "skill.md", "install.linux.python: 'requirements.txt' isn't in this folder"), (TOML, {}, "guide.md", "skill: 'guide.md' isn't in this folder"), + (TOML, {"envs/mail/skill-linux.md": None}, "skill.md", "linux.skill: 'skill-linux.md' isn't in this folder"), (f'{TOML}image = "x"\n', {}, "skill.md", "unknown key 'image'"), -], ids=["no-module", "a-driver", "install-python", "skill", "unknown-key"]) + (f'{TOML}build_settings = {{ colour = "blue" }}\n', {}, "skill.md", + "build_settings.colour: @local/~/apps/mail declares no build setting 'colour'"), +], ids=["no-module", "a-driver", "install-python", "skill", "linux-skill", "unknown-key", "build-setting"]) def test_an_app_folder_that_cant_be_read_is_refused_before_any_write(bundle_dir, toml, extra, skill, problem): _bundle(bundle_dir, toml=toml, skill=skill, **{path: data for path, data in extra.items() if data is not None}) for path in (path for path, data in extra.items() if data is None): diff --git a/tst/unit/bundle/computer_driver_env_test.py b/tst/unit/bundle/computer_driver_env_test.py index c2562fbd..da11337c 100644 --- a/tst/unit/bundle/computer_driver_env_test.py +++ b/tst/unit/bundle/computer_driver_env_test.py @@ -59,7 +59,12 @@ def press_keys(self, keys): pass LINUX = [{"os": {"type": "linux", "display": ["x11"]}, "arch": ["amd64", "arm64"]}] TOML = 'type = "computer_driver"\nmodule = "driver.py:Desk"\n' SERVE = 'services = [{ name = "driver", command = "venv/bin/python -m agentenv_protocol.computer.serve driver.py:Desk" }]' -INSTALLED = f'{TOML}port = 19000\n\n[install.linux]\nfiles = "linux"\npython = "requirements.txt"\n{SERVE}\n' +INSTALLED = (f'{TOML}port = 19000\n\n[install.linux]\nfiles = "linux"\npython = "requirements.txt"\n' + 'packages = { apt = ["xdotool", "libxtst6=2:1.2.3-1.1build1", "x11-utils/noble"] }\n' + 'sources = [{ apt = "https://ppa.example/ubuntu", suite = "noble", components = ["main"], key = "ppa.asc" }]\n' + f'downloads = [{{ url = "https://example.com/tool", sha256 = "{"a" * 64}", to = "bin/tool", arch = "amd64" }}]\n' + 'run = [{ script = "setup.sh" }, { script = "profile.sh", scope = "session" }]\n' + f'ports = [9000]\n{SERVE}\n') @pytest.fixture @@ -103,16 +108,30 @@ def test_a_driver_folder_is_written_with_its_files_and_what_its_class_declares(b assert ComputerDriverEnv.from_dict(env.to_dict()).to_dict() == env.to_dict() +#: Lines a requirements file may have: none names another file of the folder. +REQUIREMENTS = ("# what the venv installs\nmss>=9,<11 # -r is fine in a comment\n" + "--extra-index-url https://pypi.example/simple\n--find-links https://wheels.example/\n" + "--find-links \\\n https://mirror.example/wheels/\n" + "xlib @ https://example.com/python_xlib-0.33-py2.py3-none-any.whl\n--require-hashes\n") + + def test_a_driver_is_written_with_how_a_computer_installs_it_and_the_port_it_serves_on(bundle_dir): - _bundle(bundle_dir, toml=INSTALLED, **{"envs/desk/linux/requirements.txt": "mss>=9,<11\n"}) + _bundle(bundle_dir, toml=INSTALLED, **{"envs/desk/linux/requirements.txt": REQUIREMENTS, + "envs/desk/linux/ppa.asc": "key", "envs/desk/linux/setup.sh": "#!/bin/sh\n", + "envs/desk/linux/profile.sh": "#!/bin/sh\n"}) _run(bundle_dir) env = Env.get(f"{ROOT}/desk") install = env.install["linux"] - assert (env.port, install.files, install.python) == (19000, "linux", "requirements.txt") + assert (env.port, install.files, install.python, install.ports) == (19000, "linux", "requirements.txt", [9000]) + assert install.packages.apt == ["xdotool", "libxtst6=2:1.2.3-1.1build1", "x11-utils/noble"] + assert [(source.apt, source.key) for source in install.sources] == [("https://ppa.example/ubuntu", "ppa.asc")] + assert [(download.to, download.arch) for download in install.downloads] == [("bin/tool", "amd64")] + assert [(run.script, run.scope) for run in install.run] == [("setup.sh", "system"), ("profile.sh", "session")] assert [(service.name, service.scope) for service in install.services] == [("driver", "session")] - assert sorted(env.files.file_artifact_ids) == ["driver.py", "linux/requirements.txt"] + assert sorted(env.files.file_artifact_ids) == ["driver.py", "linux/ppa.asc", "linux/profile.sh", + "linux/requirements.txt", "linux/setup.sh"] assert ComputerDriverEnv.from_dict(env.to_dict()).to_dict() == env.to_dict() @@ -151,8 +170,18 @@ def test_a_driver_is_written_again_only_when_one_of_its_files_changes(bundle_dir ("left",), "install.linux.services: Value error, names a service twice: ['d', 'd']"), (f'{TOML}[install.linux]\nservices = [{{ name = "d", command = "x", scope = "root" }}]\n', LINUX, ("left",), "install.linux.services.0.scope: Input should be 'session' or 'system'"), + (f'{TOML}[install.linux]\nrun = [{{ script = "setup.sh" }}]\n{SERVE}\n', LINUX, ("left",), + "install.linux.run: 'setup.sh' isn't in this folder"), + (f'{TOML}[install.linux]\npackages = {{ apt = ["two words"] }}\n{SERVE}\n', LINUX, ("left",), + "each package is one word, as apt takes it, not ['two words']"), + (f'{TOML}[install.linux]\nsources = [{{ apt = "https://ppa.example", suite = "noble", key = "k.asc" }}]\n{SERVE}\n', + LINUX, ("left",), "sources are apt repositories to resolve packages from, and packages names none"), + (f'{TOML}[install.linux]\ndownloads = [{{ url = "https://example.com/t", sha256 = "abc", to = "t" }}]\n{SERVE}\n', + LINUX, ("left",), "install.linux.downloads.0.sha256: String should match pattern"), + (f'{TOML}[install.linux]\nports = [70000]\n{SERVE}\n', LINUX, ("left",), "a port is 1 to 65535, not [70000]"), ], ids=["no-module", "module-file", "module-class", "button", "no-platform", "unknown-key", "port", "install-os", - "install-python", "install-files", "install-escape", "no-service", "service-twice", "service-scope"]) + "install-python", "install-files", "install-escape", "no-service", "service-twice", "service-scope", + "run-missing", "apt-word", "sources-alone", "download-sha", "install-port"]) def test_a_driver_its_class_cant_be_read_from_is_refused_before_any_write(bundle_dir, toml, supports, buttons, problem): _bundle(bundle_dir, toml=toml, supports=supports, buttons=buttons) @@ -162,6 +191,20 @@ def test_a_driver_its_class_cant_be_read_from_is_refused_before_any_write(bundle Env.get(f"{ROOT}/desk") +@pytest.mark.parametrize("line", ["-r base.txt", "--requirement=base.txt", "-c pins.txt", "--constraint pins.txt", + "-e .", "--editable ./desk", "./vendored", "../elsewhere/pkg", "/opt/pkg", + "~/pkg", "file:///opt/pkg", "desk @ file:///opt/pkg", "-f wheels/", + "--find-links=wheels"]) +def test_a_requirements_file_naming_another_file_is_refused_before_any_write(bundle_dir, line): + _bundle(bundle_dir, toml=f'{TOML}[install.linux]\npython = "requirements.txt"\n{SERVE}\n', + **{"envs/desk/requirements.txt": f"mss>=9,<11\n{line} # the lock wouldn't see it\n"}) + + assert (f"install.linux.python: 'requirements.txt' line 2, {line + ' # the lock wouldn' + chr(39) + 't see it'!r}, " + "names another file, which the lock of a requirements file doesn't cover") in _problems(bundle_dir) + with pytest.raises(Exception): + Env.get(f"{ROOT}/desk") + + def test_a_dry_run_refuses_a_driver_folder_its_env_toml_doesnt_name_a_class_of(bundle_dir): _bundle(bundle_dir, toml='type = "computer_driver"\nmodule = "server.py:Desk"\n') diff --git a/tst/unit/env/envs/test_computer_env.py b/tst/unit/env/envs/test_computer_env.py index c30b077e..3fd38d25 100644 --- a/tst/unit/env/envs/test_computer_env.py +++ b/tst/unit/env/envs/test_computer_env.py @@ -4,6 +4,8 @@ and the registry are faked.""" import asyncio +import contextlib +import hashlib import importlib.metadata import io import json @@ -11,6 +13,8 @@ import re import subprocess import tarfile +import urllib.parse +from pathlib import Path from types import SimpleNamespace from unittest.mock import AsyncMock @@ -28,9 +32,10 @@ from agent_env.artifact.artifacts.file_artifact_universe import FileArtifactUniverse from agent_env.artifact.store import get_artifact_store from agent_env.cli import cli +from agent_env.config import get_config from agent_env.env import Env from agent_env.env import env as env_module -from agent_env.env.envs.computer.utils import boot, composer, gate, inside +from agent_env.env.envs.computer.utils import boot, composer, gate, inside, packages, resolve_in_base from agent_env.env.envs.computer.utils.composer import _probe from agent_env.env.envs.computer.env import ComputerEnv from agent_env.env.envs.computer.app_env import ComputerAppEnv @@ -48,6 +53,7 @@ from agent_env.store.image_store.image_store import OciRegistryImageStore from agent_env.store.image_store.local_registry_image_store import LocalRegistryImageStore from agent_env.store.image_store.oci_registry_credentials import RegistryAuth +from agent_env.store.object_store.local.store import LocalFilesystemObjectStore from agent_env.utils.docker.build import DockerBuildError from tst.unit.computer_home import Computer from tst.util.oci_registry import UBUNTU_LAYER, Registry @@ -68,7 +74,25 @@ class FakeDocker: def __init__(self, registry: Registry): self.registry, self.builds, self.fail, self.emulates, self.labels = registry, [], {}, {"linux/amd64"}, {} - self.probed, self.probe_fails = [], {} + self.probed, self.probe_fails, self.resolved = [], {}, [] + + def resolve(self, request, context, source, arch, work): + """The resolver in the base: each apt package as one .deb, each venv's requirements as one wheel.""" + self.resolved.append((arch, request)) + out, result = work / "out", {"apt": [], "wheels": {}} + for name in (re.split("[=/]", wanted)[0] for wanted in request["apt"]): + (out / "apt").mkdir(parents=True, exist_ok=True) + (out / "apt" / f"{name}_1.0_{arch}.deb").write_bytes(f"{name} for {arch}".encode()) + result["apt"].append({"file": f"{name}_1.0_{arch}.deb", "package": name, "version": "1.0", "arch": arch}) + for layer in request["python"]: + requirements = context / layer["folder"].removeprefix("/ctx/") / (layer["requirements"] or "-") + if not (requirements.is_file() and requirements.read_text().strip()) and not layer["add"]: + result["wheels"][layer["name"]] = [] + continue + (out / "wheels" / layer["name"]).mkdir(parents=True) + (out / "wheels" / layer["name"] / "mss-10.1.0-py3-none-any.whl").write_bytes(b"mss") + result["wheels"][layer["name"]] = ["mss-10.1.0-py3-none-any.whl"] + return result, out async def probe(self, links, env, driver, image): arch = self._arch(self.registry.manifests[image.rpartition("@")[2]]["config"]["digest"]) @@ -85,7 +109,9 @@ def build_image(self, dockerfile, context, tag, *, platform, labels=None, build_ self.builds.append({"platform": platform, "tag": tag, "dockerfile": dockerfile.read_text(), "files": sorted(p.relative_to(context).as_posix() for p in context.rglob("*") if p.is_file()), "conf": next(context.glob("agentenv-*.conf")).read_text(), - "confs": {conf.name: conf.read_text() for conf in context.glob("agentenv-*.conf")}}) + "confs": {conf.name: conf.read_text() for conf in context.glob("agentenv-*.conf")}, + "build_settings": {path.parent.parent.name: json.loads(path.read_text()) + for path in context.glob("apps/*/.agentenv/build_settings.json")}}) def label(self, local, platform, extra): self.labels[local] = extra @@ -121,6 +147,7 @@ def docker(local_stores, cli_routing, monkeypatch): monkeypatch.setattr(composer, "_emulated_platforms", lambda: {"linux/arm64", *fake.emulates}) monkeypatch.setattr(composer, "_installed", lambda local, layer: {"mss": "10.1.0"}) monkeypatch.setattr(composer, "_protocol_source", lambda: None) + monkeypatch.setattr(packages, "_run_resolver", fake.resolve) monkeypatch.setattr(LocalRegistryImageStore, "ensure_repository", lambda self, repository: None) return fake @@ -197,15 +224,22 @@ def test_a_computer_env_is_composed_for_each_arch_and_written_with_the_computer_ assert stored.computer_settings == {"timezone": "Europe/Paris", "theme": "dark"} arm64 = next(build for build in docker.builds if build["platform"] == "linux/arm64") assert arm64["dockerfile"].splitlines() == [ - f"FROM {base.image().image_name}", "USER root", "COPY driver/ /opt/agentenv/drivers/desk/", - "RUN python3 -m venv /opt/agentenv/drivers/desk/venv && /opt/agentenv/drivers/desk/venv/bin/pip install " - "--no-cache-dir -r /opt/agentenv/drivers/desk/requirements.txt && /opt/agentenv/drivers/desk/venv/bin/pip list " - "--format=json > /opt/agentenv/drivers/desk/installed.json", + f"FROM {base.image().image_name}", "USER root", 'ENV AGENTENV_SETTING_THEME="dark"', + "COPY driver/ /opt/agentenv/drivers/desk/", + "COPY packages/wheels/driver/ /tmp/agentenv-packages/wheels/driver/", + "RUN --network=none python3 -m venv /opt/agentenv/drivers/desk/venv && /opt/agentenv/drivers/desk/venv/bin/pip " + "install --no-index --no-deps /tmp/agentenv-packages/wheels/driver/*.whl && /opt/agentenv/drivers/desk/venv/bin/pip list " + "--format=json > /opt/agentenv/drivers/desk/installed.json && rm -rf /tmp/agentenv-packages/wheels/driver", "COPY agentenv-desk.conf /etc/supervisor/conf.d/agentenv-desk.conf", "RUN test -d /opt/agentenv/drivers/desk && test -f /etc/supervisor/conf.d/agentenv-desk.conf && test -x " "/opt/agentenv/drivers/desk/venv/bin/python", - 'ENV AGENTENV_SETTING_THEME="dark"', "USER user", f"LABEL dev.agentenv.computer-key={composed['arm64']['key']}"] - assert arm64["files"] == ["Dockerfile", "agentenv-desk.conf", "driver/driver.py", "driver/requirements.txt"] + "USER user", f"LABEL dev.agentenv.computer-key={composed['arm64']['key']}"] + assert arm64["files"] == ["Dockerfile", "agentenv-desk.conf", "driver/driver.py", "driver/requirements.txt", + "packages/wheels/driver/mss-10.1.0-py3-none-any.whl"] + assert sorted(arch for arch, _ in docker.resolved) == ["amd64", "arm64"] + assert docker.resolved[0][1]["python"] == [{"name": "driver", "folder": "/ctx/driver", + "requirements": "requirements.txt", "add": []}] + assert composed["arm64"]["lock"].startswith("sha256:") and composed["arm64"]["lock"] != composed["amd64"]["lock"] assert arm64["conf"] == ( "[program:desk-driver]\ncommand=/opt/agentenv/drivers/desk/venv/bin/python -m agentenv_protocol.computer.serve " 'driver.py:Desk\ndirectory=/opt/agentenv/drivers/desk\nenvironment=MCP_PORT="18765"\nautorestart=true\n' @@ -228,6 +262,8 @@ def test_an_env_that_differs_only_in_what_doesnt_change_the_image_reuses_the_one built, reused = (Env.get(env.id).metadata["composed"]["container"] for env in (first, second)) assert len(docker.builds) == 2 and all(reused[arch]["reused"] for arch in reused) and announced == ["first"] + assert len(docker.resolved) == 2 and {arch: r["lock"] for arch, r in reused.items()} == { + arch: r["lock"] for arch, r in built.items()} assert {arch: reused[arch]["image"] for arch in reused} == {arch: built[arch]["image"] for arch in built} assert {arch: (r["installed"], r["emulated"]) for arch, r in reused.items()} == { "amd64": ({"mss": "10.1.0"}, True), "arm64": ({"mss": "10.1.0"}, False)} @@ -236,10 +272,15 @@ def test_an_env_that_differs_only_in_what_doesnt_change_the_image_reuses_the_one @pytest.mark.parametrize("change", ["requirements", "build-setting", "service", "display-read-at-build"]) -def test_what_changes_the_image_changes_its_key_and_builds_it_again(base, driver, docker, tmp_path, change): +def test_what_changes_the_image_changes_its_key_and_builds_it_again(base, driver, docker, tmp_path, monkeypatch, + change): + staged, download = [], LocalFilesystemObjectStore.download_to_file + monkeypatch.setattr(LocalFilesystemObjectStore, "download_to_file", lambda self, url, dest: ( + staged.append(url) if packages.PREFIX in url else None, download(self, url, dest))[1]) if change == "display-read-at-build": base = _base(base.image(), config={**CONFIG, "resolution": {"default": "1280x800", "phase": "build"}}) first = Env.get(_put(base, driver).id).metadata["composed"]["container"]["arm64"]["key"] + assert staged == [] if change == "requirements": driver = _driver(tmp_path / "changed", requirements="mss>=10,<11\n") elif change == "service": @@ -251,6 +292,7 @@ def test_what_changes_the_image_changes_its_key_and_builds_it_again(base, driver "container"]["arm64"] assert again["key"] != first and again["reused"] is False and len(docker.builds) == 4 + assert bool(staged) == (change != "requirements") if display: assert 'ENV AGENTENV_SETTING_RESOLUTION="1600x900"' in docker.builds[-1]["dockerfile"] @@ -322,20 +364,20 @@ def test_a_base_built_from_its_context_is_built_here_for_each_arch_and_keyed_by_ "amd64": "unverified", "arm64": "ready"} -def test_a_checkout_composes_its_own_protocol_into_the_drivers_venv(base, driver, docker, tmp_path, monkeypatch): +def test_a_checkout_resolves_its_own_protocol_into_the_drivers_venv(base, driver, docker, tmp_path, monkeypatch): monkeypatch.setattr(composer, "_protocol_source", lambda: _protocol_checkout(tmp_path)) _put(base, driver) - dockerfile = docker.builds[0]["dockerfile"] - assert "COPY protocol /tmp/agentenv-protocol" in dockerfile - assert "pip install --no-cache-dir /tmp/agentenv-protocol -r /opt/agentenv/drivers/desk/requirements.txt" in dockerfile + assert {layer["name"]: layer["add"] for _, request in docker.resolved for layer in request["python"]} == { + "driver": ["/ctx/protocol"]} assert "protocol/pyproject.toml" in docker.builds[0]["files"] + assert "agentenv-protocol" not in docker.builds[0]["dockerfile"] def _protocol_checkout(folder): source = folder / "protocol" - (source / "src" / "agentenv_protocol").mkdir(parents=True) + (source / "src" / "agentenv_protocol").mkdir(parents=True, exist_ok=True) (source / "src" / "agentenv_protocol" / "__init__.py").write_text("") (source / "pyproject.toml").write_text("[project]\nname = 'agentenv-framework-protocol'\n") return source @@ -939,6 +981,10 @@ async def test_a_load_outside_the_home_is_refused_before_anything_is_sent(base, "theme": {"enum": ["light", "dark"], "type": "string", "default": "light"}}, "type": "object"} +MAIL_BUILD = {"properties": {"edition": {"default": "home", "enum": ["home", "pro"], "type": "string"}, + "locale": {"default": "en_US", "type": "string"}}, "type": "object"} + + #: The program of the mail app's own server. MAIL_SERVER = ("[program:app-mail]\ncommand=/opt/agentenv/apps/mail/venv/bin/python -m " "agentenv_protocol.computer.serve_app app.py:Mail\ndirectory=/opt/agentenv/apps/mail\n" @@ -948,19 +994,23 @@ async def test_a_load_outside_the_home_is_refused_before_anything_is_sent(base, def _app(base, driver, folder, compose=True, supports=None, install=None, settings=MAIL_SETTINGS, launch=None, - **changes): - """A mail app on ``base`` and ``driver``, which serves IMAP from a venv of its requirements.""" + extra=None, steps=(), build=MAIL_BUILD, linux_settings=None, **changes): + """A mail app on ``base`` and ``driver``, which serves IMAP from a venv of its requirements; ``extra`` files beside + them, by name, ``steps``, its install steps by name, scope and the OS types they run on, ``build``, the schema of + its build settings, and ``linux_settings``, that of its settings on Linux.""" folder.mkdir(parents=True, exist_ok=True) (folder / "app.py").write_text("class Mail: ...\n") (folder / "requirements.txt").write_text("agentenv-framework-protocol\n") + for name, text in (extra or {}).items(): + (folder / name).write_text(text) files = FileArtifactUniverse.put_bundled("@local/~/kit/mail__files", files={ - name: folder / name for name in ("app.py", "requirements.txt")}) + name: folder / name for name in ("app.py", "requirements.txt", *(extra or {}))}) declaration = AppDeclaration.model_validate({ "supports": supports or [{"os": {"type": "linux", "display": ["x11"]}, "arch": ["amd64", "arm64"]}], - "permissions": [], "launch": launch, "settings": settings, "skill": None, - "data": [], "install_steps": [], "ready_checks": [], - "install": {"linux": install or {"python": "requirements.txt", "services": [ - {"name": "imap", "command": "venv/bin/python imap.py"}]}}}) + "permissions": [], "settings": settings, "build_settings": build, "skill": None, + "data": [], "install_steps": [{"name": name, "scope": scope, "os": os or None} for name, scope, *os in steps], + "ready_checks": [], "os": {"linux": {"launch": launch, "settings": linux_settings, "install": install or { + "python": "requirements.txt", "services": [{"name": "imap", "command": "venv/bin/python imap.py"}]}}}}) return ComputerAppEnv.put(**{"id": "@local/~/kit/mail", "environment_name": "mail", "module": "app.py:Mail", "files": files, "declaration": declaration, "computer": base, "driver": driver, "compose": compose, **changes}) @@ -974,10 +1024,12 @@ def test_an_app_deployed_alone_is_composed_with_its_files_venv_and_programs_unde build = docker.builds[0] assert "apps/mail/app.py" in build["files"] and "apps/mail/requirements.txt" in build["files"] dockerfile = build["dockerfile"] + assert [layer for _, request in docker.resolved for layer in request["python"]][-1] == { + "name": "apps-mail", "folder": "/ctx/apps/mail", "requirements": "requirements.txt", + "add": [f"agentenv-framework-protocol[app]=={PROTOCOL_VERSION}"]} for line in ("COPY apps/mail/ /opt/agentenv/apps/mail/", - "RUN python3 -m venv /opt/agentenv/apps/mail/venv && /opt/agentenv/apps/mail/venv/bin/pip install " - f"--no-cache-dir 'agentenv-framework-protocol[app]=={PROTOCOL_VERSION}' -r " - "/opt/agentenv/apps/mail/requirements.txt", + "RUN --network=none python3 -m venv /opt/agentenv/apps/mail/venv && /opt/agentenv/apps/mail/venv/bin/pip" + " install --no-index --no-deps /tmp/agentenv-packages/wheels/apps-mail/*.whl", "RUN mkdir -p /opt/agentenv/apps/mail/.agentenv && chown user /opt/agentenv/apps/mail/.agentenv", "COPY agentenv-app-mail.conf /etc/supervisor/conf.d/agentenv-app-mail.conf"): assert line in dockerfile @@ -1000,11 +1052,192 @@ def test_an_app_that_requires_nothing_still_gets_a_venv_for_its_own_server(base, app = _app(base, driver, tmp_path / "mail", install={"services": []}, launch=launch) build = docker.builds[0] - protocol = "/tmp/agentenv-protocol[app]" if checkout else f"agentenv-framework-protocol[app]=={PROTOCOL_VERSION}" - assert (f"RUN python3 -m venv /opt/agentenv/apps/mail/venv && /opt/agentenv/apps/mail/venv/bin/pip install " - f"--no-cache-dir '{protocol}' && ") in build["dockerfile"] + protocol = "/ctx/protocol[app]" if checkout else f"agentenv-framework-protocol[app]=={PROTOCOL_VERSION}" + assert {layer["name"]: (layer["requirements"], layer["add"]) for _, request in docker.resolved + for layer in request["python"]}["apps-mail"] == (None, [protocol]) + assert "/opt/agentenv/apps/mail/venv/bin/pip install --no-index --no-deps /tmp/agentenv-packages/wheels/apps-mail/*.whl" in ( + build["dockerfile"]) assert build["confs"]["agentenv-app-mail.conf"] == MAIL_SERVER - assert boot.ready_timeout([app]) == (270.0 if at == "boot" else boot.READY_TIMEOUT_S) + assert boot.ready_timeout([app], "linux") == (270.0 if at == "boot" else boot.READY_TIMEOUT_S) + assert boot.ready_timeout([app], "windows") == boot.READY_TIMEOUT_S + + +def test_an_apps_packages_downloads_scripts_and_steps_are_fetched_at_put_and_installed_offline(base, driver, docker, + tmp_path, monkeypatch): + tool = b"#!/bin/sh\necho tool\n" + monkeypatch.setattr(packages.httpx, "stream", _serving({"https://example.com/tool": tool})) + install = {"python": "requirements.txt", "packages": {"apt": ["hello", "xdotool/noble"]}, + "sources": [{"apt": "https://ppa.example/ubuntu", "suite": "noble", "components": ["main"], + "key": "ppa.asc"}], + "downloads": [{"url": "https://example.com/tool", "sha256": hashlib.sha256(tool).hexdigest(), + "to": "bin/tool", "arch": "arm64"}], + "run": [{"script": "set up.sh"}, {"script": "profile.sh", "scope": "session"}], "services": []} + + app = _app(base, driver, tmp_path / "mail", install=install, + steps=[("index_mail", "session"), ("register_codecs", "system", "windows")], + extra={"ppa.asc": "key", "set up.sh": "mkdir -p /srv/mail\n", "profile.sh": "#!/bin/sh\n"}) + + requests = dict(docker.resolved) + assert requests["arm64"]["apt"] == ["hello", "xdotool/noble"] and requests["arm64"]["sources"] == [ + {"apt": "https://ppa.example/ubuntu", "suite": "noble", "components": ["main"], "key": "/ctx/apps/mail/ppa.asc"}] + builds = {build["platform"]: build for build in docker.builds} + lines = builds["linux/arm64"]["dockerfile"].splitlines() + first = lines.index("COPY packages/apt/ /tmp/agentenv-packages/apt/") + assert lines[first:first + 3] == [ + "COPY packages/apt/ /tmp/agentenv-packages/apt/", + "RUN --network=none apt-get install -y --no-install-recommends /tmp/agentenv-packages/apt/*.deb && rm -rf " + "/tmp/agentenv-packages/apt", "COPY driver/ /opt/agentenv/drivers/desk/"] + steps = lines.index("RUN mkdir -p /opt/agentenv/apps/mail/.agentenv && chown user /opt/agentenv/apps/mail/.agentenv") + assert lines[steps + 1:steps + 8] == [ + "RUN --network=none cd /opt/agentenv/apps/mail && './set up.sh'", + "USER user", "RUN --network=none cd /opt/agentenv/apps/mail && ./profile.sh", "USER root", + "USER user", "RUN --network=none cd /opt/agentenv/apps/mail && venv/bin/python -m " + "agentenv_protocol.computer.run_install_step app.py:Mail index_mail", "USER root"] + assert "register_codecs" not in builds["linux/arm64"]["dockerfile"] + assert {"packages/apt/hello_1.0_arm64.deb", "packages/apt/xdotool_1.0_arm64.deb", "apps/mail/bin/tool"} <= set( + builds["linux/arm64"]["files"]) + assert "apps/mail/bin/tool" not in builds["linux/amd64"]["files"] + composed = Env.get(app.id).metadata["composed"]["container"] + assert composed["arm64"]["lock"] != composed["amd64"]["lock"] + + +@pytest.mark.parametrize("failing", ["resolver", "download", "link"]) +def test_a_resolve_or_download_that_fails_fails_its_arch_at_the_resolve_stage(base, driver, docker, tmp_path, + monkeypatch, failing): + if failing == "resolver": + def refuse(request, context, source, arch, work): + raise packages.ResolveError("resolving the packages failed", "E: Unable to locate package nosuch\n") + monkeypatch.setattr(packages, "_run_resolver", refuse) + if failing == "link": + (tmp_path / "credentials").write_text("secret") + + def linking(request, context, source, arch, work): + result, out = docker.resolve(request, context, source, arch, work) + wheel = out / "wheels" / "driver" / "mss-10.1.0-py3-none-any.whl" + wheel.unlink() + wheel.symlink_to(tmp_path / "credentials") + return result, out + monkeypatch.setattr(packages, "_run_resolver", linking) + monkeypatch.setattr(packages.httpx, "stream", _serving({"https://example.com/tool": b"something else"})) + install = {"python": "requirements.txt", "services": [], + "downloads": [{"url": "https://example.com/tool", "sha256": "a" * 64, "to": "tool"}]} + + composed = Env.get(_app(base, driver, tmp_path / "mail", install=install).id).metadata["composed"]["container"] + + assert {result["stage"] for result in composed.values()} == {"resolve"} and docker.builds == [] + reason = composed["arm64"]["reason"] + if failing == "resolver": + assert reason == "resolving the packages failed: E: Unable to locate package nosuch" and composed["arm64"]["log"] + elif failing == "link": + assert reason == "the resolver's 'wheels/driver/mss-10.1.0-py3-none-any.whl' isn't a plain file it wrote" + assert not get_config().get_object_store().list(packages.PREFIX) + else: + assert reason.startswith("https://example.com/tool has the sha256 ") and reason.endswith( + f"not the {'a' * 64} its download names") + + +def test_a_stored_file_that_isnt_the_one_its_lock_names_fails_the_build_that_stages_it(base, driver, docker, + tmp_path): + lock = packages.stored(Env.get(_put(base, driver).id).metadata["composed"]["container"]["arm64"]["lock"], + "@local/~/kit/desktop") + sha256 = lock.wheels["driver"][0].sha256 + store, prefix = packages._store("@local/~/kit/desktop") + Path(urllib.parse.urlparse(store.object_url(f"{prefix}/files/{sha256}")).path).write_bytes(b"tampered") + + again = Env.get(_put(base, driver, computer_settings={"theme": "dark"}).id).metadata["composed"]["container"]["arm64"] + + assert (again["status"], again["stage"]) == ("failed", "build") + assert again["reason"].endswith(f"agentenv-packages/files/{sha256} isn't the file the lock names: its sha256 is " + f"{hashlib.sha256(b'tampered').hexdigest()}") + + +def test_a_venv_whose_requirements_resolve_to_nothing_is_made_empty(base, docker, tmp_path): + driver = _driver(tmp_path / "empty", requirements="") + + _put(base, driver) + + lines = docker.builds[0]["dockerfile"].splitlines() + assert "RUN --network=none python3 -m venv /opt/agentenv/drivers/desk/venv && /opt/agentenv/drivers/desk/venv/bin/pip " \ + "list --format=json > /opt/agentenv/drivers/desk/installed.json" in lines + assert not [line for line in lines if "agentenv-packages" in line] + + +def test_two_puts_resolving_the_same_inputs_at_once_both_get_the_first_stored_lock(docker, tmp_path, monkeypatch): + folder = tmp_path / "ctx" / "driver" + folder.mkdir(parents=True) + (folder / "requirements.txt").write_text("mss\n") + wants = [packages.Want("driver", Install(python="requirements.txt", services=[Service(name="d", command="x")]), True)] + inputs = packages.inputs_digest(wants, tmp_path / "ctx", "base", "arm64") + first, _ = packages.resolve(inputs, wants, tmp_path / "ctx", "base", "arm64", "@local/~/kit/desktop", tmp_path / "1") + + def newer(request, context, source, arch, work): + result, out = docker.resolve(request, context, source, arch, work) + (out / "wheels" / "driver" / "mss-10.1.0-py3-none-any.whl").write_bytes(b"mss, published again") + return result, out + monkeypatch.setattr(packages, "_run_resolver", newer) + second, _ = packages.resolve(inputs, wants, tmp_path / "ctx", "base", "arm64", "@local/~/kit/desktop", + tmp_path / "2") + monkeypatch.setattr(packages.os, "getuid", lambda: 4242) + + assert second == first == packages.stored(inputs, "@local/~/kit/desktop") + assert packages.inputs_digest(wants, tmp_path / "ctx", "base", "arm64") == inputs + + +def test_the_resolver_installs_the_apt_packages_it_downloads_before_building_wheels(tmp_path, monkeypatch): + ran = [] + monkeypatch.setattr(resolve_in_base, "KEYRINGS", tmp_path / "keyrings") + monkeypatch.setattr(resolve_in_base, "SOURCES", tmp_path) + monkeypatch.setattr(resolve_in_base.subprocess, "run", lambda argv, **kwargs: ran.append(argv) or SimpleNamespace( + stdout="")) + monkeypatch.setattr(resolve_in_base.os, "chown", lambda *args: None) + request = {"apt": ["build-essential", "libxml2-dev"], "sources": [], "owner": [0, 0], "python": [ + {"name": "driver", "folder": str(tmp_path), "requirements": "requirements.txt", "add": []}]} + (tmp_path / "request.json").write_text(json.dumps(request)) + + resolve_in_base.main(str(tmp_path / "request.json"), str(tmp_path / "out")) + + commands = [" ".join(argv[:3]) for argv in ran] + assert commands[:3] == ["apt-get update -q", "apt-get install -y", "apt-get install -y"] + assert "--download-only" in ran[1] and "--download-only" not in ran[2] and ran[2][-2:] == request["apt"] + assert ran[-1][1:3] == ["wheel", "--no-cache-dir"] + + +def _serving(bodies): + """``httpx.stream`` answering each URL with its body.""" + @contextlib.contextmanager + def stream(method, url, **kwargs): + yield httpx.Response(200, content=bodies[url], request=httpx.Request(method, url)) + return stream + + +def test_an_apps_build_settings_are_fixed_in_its_folder_and_a_change_builds_another_image_from_the_same_lock( + base, driver, docker, tmp_path): + first = _app(base, driver, tmp_path / "mail", build_settings={"locale": "fr_FR"}) + second = _app(base, driver, tmp_path / "mail", build_settings={"locale": "fr_FR", "edition": "pro"}) + + assert [build["build_settings"] for build in docker.builds if build["platform"] == "linux/arm64"] == [ + {"mail": {"locale": "fr_FR"}}, {"mail": {"edition": "pro", "locale": "fr_FR"}}] + keys = {Env.get(app.id, app.version).metadata["composed"]["container"]["arm64"]["key"] for app in (first, second)} + assert len(keys) == 2 and len(docker.resolved) == 2 + stored = Env.get(second.id) + assert stored.build_settings == {"locale": "fr_FR", "edition": "pro"} + assert ComputerAppEnv.from_dict(stored.to_dict()).to_dict() == stored.to_dict() + + +@pytest.mark.parametrize("build, values, problem", [ + ({**MAIL_BUILD, "required": ["locale"]}, {}, "build_settings: 'locale' is a required property"), + (MAIL_BUILD, {"colour": "blue"}, "build_settings.colour: @local/~/kit/mail declares no build setting 'colour'"), + (MAIL_BUILD, {"edition": "office"}, "build_settings.edition: 'office' is not one of ['home', 'pro']"), + (None, {"edition": "pro"}, "build_settings.edition: @local/~/kit/mail declares no build setting 'edition'"), +], ids=["required", "unknown", "enum", "none-declared"]) +def test_build_settings_an_app_doesnt_declare_or_its_schema_refuses_are_refused_before_anything_is_stored( + base, driver, docker, tmp_path, build, values, problem): + with pytest.raises(ValueError, match=re.escape(problem)): + _app(base, driver, tmp_path / "mail", build=build, build_settings=values) + + assert docker.builds == [] + with pytest.raises(NotFoundError): + Env.get("@local/~/kit/mail") def test_an_app_is_composed_only_for_the_arches_it_supports(base, driver, docker, tmp_path): @@ -1086,6 +1319,19 @@ async def test_a_load_whose_connection_to_the_driver_breaks_says_so(base, driver await env.load_file_artifact_universe(_files(tmp_path, **{"a.txt": b"aaaa"})) +def test_a_deploys_app_settings_are_checked_against_its_settings_on_its_computers_os(base, driver, docker, tmp_path): + linux = {**MAIL_SETTINGS, "properties": {**MAIL_SETTINGS["properties"], "font": {"type": "string", "default": ""}}} + app = _app(base, driver, tmp_path / "mail", linux_settings=linux) + + def saved(values): + return DeployEnvTaskStep(id="d", version=None, env_id=app.id, + env_params={"app_settings": {"mail": values}}).preflight() + + assert saved({"font": "Mono"}) == [] and (app.settings_schema("linux"), app.settings_schema("windows")) == ( + linux, MAIL_SETTINGS) + assert "env_params.app_settings.mail.font: 3 is not of type 'string'" in saved({"font": 3})[0] + + @pytest.mark.asyncio @pytest.mark.parametrize("values, problem", [ ({"colour": "blue"}, "env_params.app_settings.mail.colour: @local/~/kit/mail declares no setting 'colour'"), @@ -1093,7 +1339,9 @@ async def test_a_load_whose_connection_to_the_driver_breaks_says_so(base, driver ({"theme": "blue"}, "env_params.app_settings.mail.theme: 'blue' is not one of ['light', 'dark']"), ({"password": "hunter2"}, "env_params.app_settings.mail.password is a secret, so it takes a secret:KEY"), ({"token": "hunter2"}, "env_params.app_settings.mail.token is a secret, so it takes a secret:KEY"), -], ids=["unknown", "type", "enum", "secret-value", "optional-secret-value"]) + ({"edition": "pro"}, "env_params.app_settings.mail.edition is one of @local/~/kit/mail's build settings, which only " + "another image changes"), +], ids=["unknown", "type", "enum", "secret-value", "optional-secret-value", "build-setting"]) async def test_app_settings_an_app_doesnt_declare_are_refused_at_save_and_at_deploy(base, driver, docker, boots, tmp_path, values, problem): app = _app(base, driver, tmp_path / "mail") diff --git a/tst/unit/serialization/golden/env/computer_app.json b/tst/unit/serialization/golden/env/computer_app.json index 109098b9..5e4d0e19 100644 --- a/tst/unit/serialization/golden/env/computer_app.json +++ b/tst/unit/serialization/golden/env/computer_app.json @@ -8,25 +8,27 @@ "data/add", "data/get" ], - "install": { + "install_steps": [], + "os": { "linux": { - "files": ".", - "python": "requirements.txt", - "services": [ - { - "command": "venv/bin/python serve.py", - "name": "notes" - } - ] + "install": { + "files": ".", + "python": "requirements.txt", + "services": [ + { + "command": "venv/bin/python serve.py", + "name": "notes" + } + ] + }, + "launch": { + "at": "demand", + "command": [ + "notes" + ] + } } }, - "install_steps": [], - "launch": { - "at": "demand", - "command": [ - "notes" - ] - }, "permissions": [], "ready_checks": [], "settings": { diff --git a/tst/unit/serialization/test_golden_wire_format.py b/tst/unit/serialization/test_golden_wire_format.py index 802d442a..94786b8d 100644 --- a/tst/unit/serialization/test_golden_wire_format.py +++ b/tst/unit/serialization/test_golden_wire_format.py @@ -183,9 +183,9 @@ def _website_env() -> WebsiteEnv: files=ARTIFACT_FIXTURES["file_artifact_universe"](), declaration=AppDeclaration.model_validate({ "supports": [{"os": {"type": "linux", "display": ["x11"]}, "arch": ["amd64"]}], "permissions": [], - "install": {"linux": {"files": ".", "python": "requirements.txt", - "services": [{"name": "notes", "command": "venv/bin/python serve.py"}]}}, - "launch": {"at": "demand", "command": ["notes"]}, + "os": {"linux": {"install": {"files": ".", "python": "requirements.txt", + "services": [{"name": "notes", "command": "venv/bin/python serve.py"}]}, + "launch": {"at": "demand", "command": ["notes"]}}}, "settings": {"properties": {"signature": {"default": "", "title": "Signature", "type": "string"}}, "title": "Settings", "type": "object"}, "skill": {"file": "skill.md"}, "data": ["data/add", "data/get"], "install_steps": [], "ready_checks": []}),